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(57) A BD-ROM stores PlayList information. The 
PlayList information defines a playback section of each 
of a plurality of AV clips and includes Main Path informa- 
tion and SubPath information. The Main Path information 
designates one of the AV clips as a Main Clip and defines 
a portion of the Main Clip as a primary playback section. 
The SubPath information designates another one of the 



AV clips as a SubClip and defines a portion of the SubClip 
as a secondary playback section that is to be played back 
in synchronism with the primary playback section. The 
BD-ROM stores, the one of the AV clips designated as 
the SubClip along with an EP map. The EP map shows 
a plurality of entry points on the SubClip in a one-to-one 
correspondence with entry times on the SubClip timeline. 
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Description 

Technical Field 

[0001 ] The present invention relates to the field of syn- 
chronous application technology. 

Background Art 

[0002] Synchronous application technology defines 
synchronous playback of a plurality of digital streams 
stored on different recording media, so that it appears to 
users to be playback of a single movie. 
Synchronous applications are composed of different 
types of digital streams including main streams and sub- 
streams. A main stream is a digital stream containing 
high-quality video, whereas a substream is a digital 
stream without such high-quality video. Main streams are 
distributed to users via large-volume optical discs typified 
by BD-ROMs storing the main streams. On the other 
hand, substreams are distributed to users via, for exam- 
ple, the Internet. 

[0003] Since digital streams constituting a single mov- 
ie can be distributed to users via different media, more 
flexibility is offered in production of variations of the mov- 
ie. Thus, wider variations can be provided to playback of 
a single movie. 

Note that the following patent literature discloses prior 
art relating to synchronous application technology. Pat- 
ent Literature 1 : 

JP Patent Application Publication No. 2002-247526 
Disclosure of the Invention 

[Problems the Present Invention is going to Solve] 

[0004] In order to ensure playback by consumer home 
appliances, a synchronous application needs to have a 
mechanism for allowing a consumer home appliance to 
execute trick play. The term "trick play" refers to such 
playback operations as fast-forwarding, rewinding, chap- 
ter-search, and time-search. Those playback operations 
are implemented using the "random access" function to 
a digital stream. With the random access function, an 
access is made to a point on the digital stream corre- 
sponding to any desired point on the timeline of the digital 
stream. Inthecaseof asynchronous application, random 
access to the main stream is not sufficient to execute 
trick play. It is also necessary that the substream can be 
accessed at random. 

[0005] Substreams may carry various kinds of play- 
back data, such as audio, graphics, and standard-quality 
video. Generally, a main stream containing high-quality 
video is composed of GOPs (Group of Pictures), which 
are independently decodable units. However, a sub- 
stream may or may not be composed of data in any units, 
such as GOPs. Even if a substream is composed of data 
in some units, which are independently decodable, the 



display rate, sampling frequency, and bit rate of such 
units may greatly vary from stream to stream. 
[0006] Since substreams differ from one another in dis- 
play rate, sampling frequency, bit rate, and independently 
5 decodable units, random access to some substream may 
not be made as fast as random access to the main 
stream. Thus, playback of a substream may not be start- 
ed immediately upon request and may be notably de- 
layed. 

10 [0007] A synchronous application is designed for syn- 
chronous playback of a main stream and a substream. 
Thus, delay in playback start of the substream causes 
playback start of the main stream to be delayed as well. 
As a result, there is a considerable delay in response to 

15 a user request for playback. 

The present invention aims to provide a recording medi- 
um and a playback device for avoiding a response delay 
occurring when random access to a substream is made 
in synchronism with random access to the main stream. 

20 

[Means to Solve the Problems] 

[0008] In an attempt to achieve the above aim, the 
present invention provides a recording medium having 
25 playlist information recorded thereon. The playlist infor- 
mation defines a playback section of each of a plurality 
of digital streams, and includes main-path information 
and sub-path information. The main-path information 
designates one of the digital streams as a main stream, 
30 and defines a portion of the main stream as a primary 
playback section. The sub-path information designates 
another one of the digital streams as a substream, and 
defines a portion of the substream as a secondary play- 
back section that is to be synchronously played back with 
35 the primary playback section. The recording medium fur- 
ther has recorded thereon the one of the plurality of digital 
streams designated as the substream, together with an 
entry map. The entry map indicates a plurality of entry 
points on the substream in one-to-one correspondence 
with a plurality of entry times on a timeline of the sub- 
stream. 

[Effects of the Invention] 

45 [0009] Since an entry map is provided for a digital 
stream which is a substream, random access to any de- 
sired point on the substream is ensured at high speed 

using the entry map. 

The PlayUst information defines synchronization of the 
50 main stream and the substream on precondition that 
high-speed random access is possible for both the 
streams. This arrangement allows a playback device to 
execute skip playback and fast-speed playback of a syn- 
chronous application, which requires synchronous play- 
55 back of the two streams, immediately upon a user oper- 
ation. 

[001 0] This advantage will lead to the widespread use 
of applications made of main streams and substreams 
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in combination. 

Since high-speed random access to a substream not 
containing GOPs is ensured, playback of a movie com- 
posed of a main stream and such a substream can be 
started promptly from any point on the timeline. That is, 
a movie composed of a main stream and a substream 
can be processed in the same manner as a movie com- 
posed solely of a main stream. Consequently, a wider 
and richer variety is offer to playback of a single movie 
through a combination of a main stream with a sub- 
stream. 

[0011] Here, it should be noted that entry maps pro- 
vided for substreams naturally differ from one another in 

time accuracy. 

Since the time accuracy of each entry map differs de- 
pending on the data carried by a substream associated 
with the entry map, no information is available regarding 
the time interval between adjacent entry times and what 
kind of data points are designated as entry times. With 
this being a situation, it is unknown to what extent the 
stream analysis needs to be performed. Without infor- 
mation regarding the extent of required stream analysis, 
it cannot be ensured that random access to the sub- 
stream is made as fast as random access to the main 
stream. In other words, there is an undesirable possibility 
that playback of the substream cannot be started timely 
and will be delayed notably. 

[0012] To eliminate the above undesirable possibility, 
the entry map may be of a first type or a second type. 
The first type entry map indicates the plurality of entry 
times at a constant time interval on the timeline of the 
substream or the plurality of entry points at a constant 
data interval on the substream. The second type entry 
map indicates the entry points each located at a head of 
a complete data set. The entry map includes a flag indi- 
cating whether the entry map is of the first type or the 
second type. 

[0013] According to the above structure, in the case 
where the flag indicates the first type, the entry points 
exist either at constant time intervals or at constant data 
intervals. Thus, with reference to the flag, the playback 
device is informed thatstream analysis, if required, needs 
to cover at most the data worth the constant time interval 
or the constant data interval. That is, the playback device 
is informed that the requested access point is specified 
by analyzing, even in the worst-case conditions, data that 
is worth the constant time interval or the constant data 
interval. 

[0014] In the case where the flag indicates the second 
type, the entry points are located at the heads of complete 
data sets. Thus, with reference to the flag, the playback 
device is informed that data read shall be started from 
any of the entry points, which are located at inconstant 
intervals. As a result, it is ensured that playback is duly 
started from any desired playback point. 
[0015] Since the flag causes the playback device to 
judge whether stream analysis of data worth at most a 
predetermined time or data interval is required or no 



stream analysis is required at all. Thus, even if not only 
the Main Clip but also the SubClip needs to be accessed 
at random, the playback device is not subject to exces- 
sive burden. As a result of the reduced burden, response 

5 of the playback device to a user operation improves. 

Brief Description of The Drawings 
[0016] 

10 

Fig. 1 illustrates a form of using a recording medium 

according to the present invention; 

Fig. 2 is illustrates the internal structure of a BD- 

ROM; 

^5 Fig. 3 schematically illustrates the structure of a file 
with the extension ".m2ts"; 

Fig. 4 illustrates the processes through which TS 
packets constituting a Main Clip are recorded onto 
the BD-ROM; 

20 Fig. 5 illustrates the internal structure of a video 
stream used to present a movie; 
Fig. 6A illustrates the internal structure of an IDR 
picture. 



25 [0017] 

Fig. 68 illustrates the internal structure of a Non-IDR 
I picture, and Fig. 6C illustrates the dependency re- 
lationship between the Non-IDR I picture and other 
30 pictures; 

Fig. 7 illustrates the processes through which an IDR 
picture and a Non-IDR I picture are converted into 
TS packets; 

Fig. 8 illustrates the internal structure of Clip infor- 
ms mation; 

Fig. 9 illustrates the setting of an EP_map for a video 
stream carrying a movie (application_type = 1); 
Fig. 1 0 shows pairs of EP_Low and EP_High values 
representing the PTS_EP_start and the SPN_EP_ 
^0 start of entry points #1 - #7 illustrated in Fig. 9; 

Fig. 11 illustrates the internal structure of a local stor- 
age; 

Fig. 1 2 illustrates the internal structures of a primary 

audio stream and a secondary audio stream; 
45 Fig. 13 illustrates the internal structure of a PG 
stream; 

Fig. 14 illustrates the internal structure of an IG 

stream; 

Fig. 15 illustrates the data structure of Clip informa- 
50 tion stored on the local storage; 

Fig. 16 illustrates an EP_map for a primary or sec- 
ondary audio stream; 

Fig. 17 illustrates an EP_map set on the PG stream 
timeline; 

55 Fig. 18 illustrates an EP_map set on the IG stream 
timeline; 

Fig. 1 9 illustrates the data structure of PlayList infor- 
mation; 
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Fig. 20 illustrates the relationship between the Main 
Clip and the PlayList information; 
Fig. 21 illustrates the internal structure of PlayList- 
Mark information in PlayList information; 
Fig. 22 illustrates chapter positions specified by the 
PLMark information included in the PlayList informa- 
tion; 

Fig. 23 illustrates the internal structure of SubPath 
information in detail; 

Fig. 24 illustrates the relationship among the Main 

Clip stored on the BD-ROM and the SubClip and the 

PlayList information both stored on the local storage; 

Fig. 25 illustrates the EP map for the Main Clip, a 

Playltem timeline, the EP_map for the SubClip, 

which is either a primary or secondary audio stream, 

and a SubPlayltem timeline; 

Fig. 26 illustrates the EP_map for the Main Clip, the 

Playltem timeline, the EP_map for the SubClip, 

which is either a PG or IG stream; 

Fig. 27 shows a table of permissible combinations 

of the values of the SubPath_type, the application_ 

type, and the EP stream type; 

Fig. 28 illustrates the virtual file system created by a 

playback device 300; 

Fig. 29 illustrates the internal structure of the play- 
back device according to the present invention; 
Fig. 30 is a flowchart illustrating the processing steps 
for executing jump playback; 
Fig. 31 schematically illustrate how a random access 
point is determined using the EP_map illustrated in 
Fig. 25; 

Fig. 32 is a flowchart illustrating the processing steps 

for converting a coordinate TM on the Main Clip and 

the SubClip to a corresponding address; 

Fig. 33 illustrates the relationship among variables 

k and h and the random access point in the case 

where the SubClip is a primary or secondary audio 

stream; 

Fig. 34 illustrates the relationship among the varia- 
bles /cand h and the random access point in the case 
where the SubClip is a PG or IG stream; 
Fig. 35 illustrates one example of PiP playback; 
Fig. 36A illustrates an HD image and an SD image 
for comparison, and 

[0018] 

Fig. 36B illustrates how the secondary video is 
scaled up and down; 

Fig. 37 illustrates the contents stored on the local 
storage according to a second embodiment; 
Fig. 38 illustrates the internal structure of clip infor- 
mation stored on the local storage according to the 
second embodiment; 

Fig. 39 illustrates an EP_map for the secondary vid- 
eo stream in the same manner to Fig. 9; 
Fig. 40 illustrates PlayList information defining a syn- 
chronous application implementing statically syn- 



chronized PiP playback; 

Fig. 41 illustrates, in the same manner as Figs. 25 
and 26, how the PlayList information defines the syn- 
chronization between the Main Clip containing pri- 

5 mary video and the SubClip containing secondary 

video; 

Figs. 42A-42C illustrate a dynamically synchronized 
PiP application; 

Fig. 43 illustrates the internal structure of the PlayList 
10 information defining dynamically synchronized PiP 
playback; 

Fig. 44 illustrates the internal structure of a playback 
device according to the second embodiment; 
Fig. 45 is a flowchart illustrating the processing steps 
15 for executing PL playback; 

Fig. 46 schematically illustrates, in the same manner 
as Fig. 31 , random to the Main Clip and to the Sub- 
Clip; and 

Fig. 47A illustrates the playback control for executing 
20 dynamically synchronized PiP playback, 

[0019] 

Fig. 478 illustrates dynamically synchronized PiP 
25 playback executed in the case where a rock point 
once passed is reached again in a normal playback 
mode as a result of rewinding, and Fig. 47C illus- 
trates PiP playback executed in the case where a 
playback section of the secondary video is located 
30 at some point subsequent to the end point of a play- 
back section of the primary video. 

[Description of Reference Numerals] 



35 [0020] 
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24 Composition Buffer 

25 Composition Controller 

26 Interactive Graphics Plane 

27 GLUT Unit 

28 Compositor 

29 Compositor 

30 Switch 

31 Network Device 

32 Switch 

33 Arrival Time Clock Counter 

34 Source De-Packetizer 

35 PID Filter 

36 Switch 

37 Transport Buffer 

38 Elementary Buffer 

39 Audio Decoder 

40 Transport Buffer 

41 Buffer 

42 Audio Decoder 

43 Mixer 

44 Scenario l\/lemory 

45 Controller 

46 Main Exchanger 

47 Sub-Exchanger 

48 PL Playback Controller 

49 PSR Set 

Best Mode for Carrying Out the Invention 
(FIRST EMBODIMENT) 

[0021] The following describes an embodiment of a 
recording medium according to the present invention. 
First of all, among various acts of practicing a recording 
medium of the present invention, an act of using is de- 
scribed. Fig. 1 illustrates a form of using the recording 
medium according to the present invention. In Fig. 1, a 
local storage 200 is the recording medium according to 
the present invention. The local storage 200 is a hard 
disk built into a playback device 300. The local storage 
200 is used with a BD-ROM 100 to together supply a 
movie to a home theater system composed of the play- 
backdevice 300, a remote controller 400, and a television 
set 500. 

[0022] The local storage 200 is a built-in hard disk used 
to store contents supplied from movie distributors. 
The playback device 300 is a network-capable digital 
home appliance and used to play back the BD-ROM 1 00. 
In addition, the playback device 300 can expand the func- 
tionality of the BD-ROM 100 through the combined use 
of the contents stored on the BD-ROM 1 00 with contents 
downloaded from a movie distributor's server via the net- 
work. 

[0023] The remote controller 400 is used to receive 
user input specifying, for example, a chapter to be played 
back or time to start playback. 

The television set 500 displays playback produced by 
the playback device 300. 



The contents stored on the local storage 200 are used 
in combination with the contents stored on the BD-ROM 
1 00. Here, the contents stored on the local storage 200 
and not physically present on the BD-ROM 1 00 are proc- 

5 essed as if they are stored on the BD-ROM. This tech- 
nique is called "virtual file system". 
[0024] The recording medium according to the present 
invention is designed for such a use in combination with 
the BD-ROM 100. This concludes the description of the 

10 act of using the recording medium of the present inven- 
tion. 

Now, the following sequentially describes the internal 
structures of the BD-ROM 100 and of the local storage 

200. 

15 <Overview of BD-ROM> 

Fig. 2 illustrates the internal structure of the BD-ROM. In 
the figure, the BD-ROM is illustrated on the fourth level, 
and the track of the BD-ROM is illustrated on the third 
level. In the figure, the track is laterally stretched out, 

20 although the track in practice spirals outwards from the 
center of the BD-ROM. The track is composed of a lead- 
in area, a volume area, and a lead-out area. The volume 
area has a layer model of a physical layer, a file system 
layer, and an application layer. The first level illustrates, 

25 in a directory structure, a format of the application layer 
(application format) of the BD-ROM. As illustrated on the 
first level, the BD-ROM has a ROOT directory, and the 
ROOT directory has a BDMV directory. 
[0025] The BDMV directory has three subdirectories 

30 called a PLAYLIST directory, a CLIPINF directory, and 
a STREAM directory. 

The STREAM directory stores files with the extension 
".m2ts" (e.g. a file called 00001. m2ts) containing data 
constituting the main body of a digital stream. 

35 The PLAYLIST subdirectory stores files with the exten- 
sion ".mpis" (e.g. a file called 00001 .mpis). 
[0026] The CLIPINF directory stores files with the ex- 
tension ".dpi" (e.g. a file called 00001 .dpi). 
The following describes an AV clip. Clip Information, and 

"^0 PlayList information, which are components of the BD- 
ROM. 

<BD-ROM Structure: 1. AV Clip> 

First of all, a description is given of files with the extension 

".m2ts". Fig. 3 schematically illustrates the structure of a 

45 file with the extension ".m2ts". Each file with the extension 
".m2ts" (namely, 00001. m2ts, 00002.m2ts, 
00003. m2ts, ...) contains an AV clip. The AV clip (illus- 
trated on the fourth level) is created as follows. A video 
stream containing a plurality of video frames (pictures 

50 pjl,pj2, pj3, ...) and an audio stream containing a plurality 
of audio frames (both illustrated on the first level) are 
separately converted to PES packets (illustrated on the 
second level), and further converted to TS packets (illus- 
trated on the third level). These TS packets of the video 

55 and audio streams are multiplexed to form the AV clip. 
As in the example illustrated in Fig. 3, anAV clip contain- 
ing video is specifically referred to as a "Main Clip" to 
differentiate it from other AV clips not containing video. 
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[0027] Next, a description is given of Inow a IVlain Clip, 
whicin is a digital stream of the MPEG2-TS format, is re- 
corded on the BD-ROM. Fig. 4 illustrates the processes 
through which TS packets constituting the Main Clip are 
recorded onto the BD-ROM. In Fig. 4, the TS packets of 
the Main Clip are illustrated on the first level. 
Each TS packet of the Main Clip is 1 88 bytes in length. 
As Illustrated on the second level, "TS_extra_header" 
(denoted as "EX" In the figure) is attached to each TS 
packet, so that the packet length becomes 1 92 bytes. 
[0028] The third and fourth levels illustrate the physical 
units of the BD-ROM in relation to the TS packets. As 
Illustrated on the fourth level, the track on the BD-ROM 
is divided into a plurality of sectors. The TS packets with 
the TS_extra_header (hereinafter, simply "EX-TS pack- 
ets") are divided into groups of 32 EX-TS packets, and 
each group is written Into three sectors of the BD-ROM. 
Each group composed of 32 EX-TS packets amounts to 
6,144 (= 32 X 192) bytes, which is equal to the total size 
of three sectors (= 2048 x 3). Each group of 32 EX-TS 
packets stored In three sectors of the BD-ROM Is referred 
to as an "Aligned Unit". When data is recorded onto the 
BD-ROM, encryption of data is carried out in Aligned 
Units. 

[0029] As Illustrated on the fifth level, an error correc- 
tion code Is Inserted every 32 sectors to constitute an 

ECC block. As long as accessing the BD-ROM in Aligned 
Units, a playback device is ensured to obtain a complete 
set of 32 EX-TS packets without missing any part. This 
completes the description of the processes of recording 
the Main Clip onto the BD-ROM. 
<Description of AV Clip: 1 . Video Stream> 
The following describes the Internal structure of a video 
stream. 

[0030] Fig. 5 illustrates the internal structure of a video 
stream used to present a movie. The video stream illus- 
trated in Fig. 5 is composed of a plurality of pictures 
aligned In the order of coding. 

In the figure, the reference numerals "I", "P", and "B" 
denote an I picture, B picture, and P picture, respectively. 
There are two types of I pictures, one is an IDR picture 
and the other Is a Non-1 DR I picture. Non-1 DR I pictures, 
P pictures, and B pictures are coded using correlation 
with other pictures. More specifically, a B picture is com- 
posed of Bidirectioanlly Predictive (B) slices. A P picture 
Is composed of Predictive (P) slices. There are two types 
of B pictures, one Is a reference B picture, and the other 
is a nonreference B picture. 

[0031] In Fig. 5, a Non-IDR I picture is denoted as "I", 
and an IDR picture Is denoted as "IDR". The same de- 
notations are used throughout the following descriptions. 
This concludes the description of the video stream used 
to present a movie. 

Next, the following describes the internal structures of an 
IDR picture and a Non-IDR I picture. Fig. 6A Illustrates 
the Internal structure of an IDR picture. As illustrated In 
the figure, the IDR picture is composed of a plurality of 
Intra-type slices. Fig. 6B illustrates the internal structure 



of a Non-IDR I picture. Different from the IDR picture 
composed solely of Intra-type slices, the Non-IDR I pic- 
ture Is composed of slices of Intra-type, P-type, and B- 
type. Fig. 6C Illustrates the dependency relationship be- 
5 tween the Non-IDR I picture and other pictures. A Non- 
IDR I picture may be composed of B and P slices and 
thus may have a dependency relationship with other pic- 
tures. 

<Description of AV Clip: 2. Recording to BD-ROM> 

10 Next, the following describes how IDR pictures and Non- 
IDR I pictures are converted into TS packets and record- 
ed onto the BD-ROM. Fig. 7 Illustrates the processes 
through which an IDR or Non-IDR I picture Is converted 
into TS packets. In the figure, the first level illustrates an 

15 IDR or Non-IDR I picture. The second level illustrates an 
Access Unit stipulated according to MPEG4-AVC. A plu- 
rality of slices constituting the IDR or Non-IDR I picture 
is sequentially aligned. Then, AUD (Access Unit Delim- 
iter), SPS (Sequence Parameter Set), PPS (Picture Pa- 

20 rameter Set), SEI (Supplemental Enhanced Information) 
are attached to the slice sequence. In this way, the picture 
slices are converted into an Access Unit. 
[0032] AUD, SPS, PPS, SEI, Access Unit mentioned 
above are information all stipulated according to MPEG4- 

25 AVC and described In various documents, such as "ITU- 
T Recommendation H.264". For the details, such docu- 
ments should be referenced. The point in this description 
is that AUD, SPS, PPS, and SEI need to be supplied to 
a playback device for allowing random access to the vid- 

30 eo stream. 

The third level represents NAL units. AUD, SPS, PPS, 
SEI, and each slice illustrated on the second level are 
each attached with a header so as to be converted into 
separate NAL units. NAL units are units supported by the 

35 Network Abstraction Layer (NAL) stipulated according to 
MPEG-4 AVC and described in various documents, such 
as "ITU-T Recommendation H.264". For the details, such 
documents should be referenced. The point In this de- 
scription Is that AUD, SPS, PPS, SED, and each slice 

"^0 are converted into separate NAL units and manipulated 
independently in the Network Abstraction Layer. 
[0033] As described above, a plurality of NAL units ob- 
tained by converting the single picture are further con- 
verted into PES packets illustrated on the fourth level. 

45 The PES packets are then converted into TS packets. 
Finally, the resulting TS packets are recorded onto the 
BD-ROM. 

In order to play back one GOP, the decoder needs to be 
supplied with a NAL unit containing AUD, among NAL 

50 units constituting the first IDR or Non-IDR I picture In the 
GOP. That is, the NAL unit containing AUD Is used as 
an index for decoding the IDR picture or Non-IDR I pic- 
ture. In this embodiment, each NAL unit containing AUD 
is regarded as a point. For playback of the video stream, 

55 the playback device recognizes each of such a point as 
an entry point for playback of a Non-IDR I picture and an 
I DR picture. Consequently, for implementation of random 
access to the Main Clip, it is extremely important for the 
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playback device to recognize tlie locations of AUDs of 
IDR pictures and Non-1 DR I pictures. This concludes the 
description of the structure of the MPEG-4AVC video 
stream used for presenting a movie. 
<BD-ROI\/l Structure: 2. Clip information> 
Next, the following describes files with the extension 
".dpi". Each file with the extension ".dpi" (e.g. 00001 .dpi, 
00002.clpi, OOOOS.clpi,...) contains Clip information. 
Each piece of Clip information is management informa- 
tion of an individual Main Clip. Fig. 8 illustrates the internal 
structure of a piece of Clip information. As illustrated in 
the left block of the figure, the Clip information is com- 
posed of the following fields: 

(i) "Cliplnfo ( )" storing the attributes of the AV clip file; 

(ii) "Sequence lnfo( )" storing information related to 
the ATC sequence and STC Sequence; 

(ill) "Program lnfo( )" storing information related to 

the program sequence; and 

(iv) "Characteristics Point Info (i.e., CPI ( ))". 

<Description of Clip information: 1. Cliplnfo 
First, a description of Cliplnfo will be given. In the figure, 
leader lines ctl indicates that the structure of Cliplnfo() 
is illustrated in greater detail. As indicated by the leader 
lines ctl, the Cliplnfo ( ) is composed of the following 
fields :"clip_stream_type" indicating the type of the digital 
stream; "application_type" indicating the type of an ap- 
plication that the Main Clip constitutes; and "TS_record- 
ing rate" indicating the data rate of the Main Clip. The 
application_type is set to the value "1" to indicate that 
the associated Main Clip constitutes a Movie application. 
This concludes the description of Cliplnfo( ). 
Next, the following describes Cliplnfo. 
[0034] The Cliplnfo includes as many "EP_map_for_ 
one_stream[0] - [Ne - 1 ]" fields as the value held by "Ne", 
and also includes the following attributes of each "EP_ 
map_for_one_stream". The attributes of each "EP_map_ 
for one stream" include: "stream_PID[0] - [Ne - 1]" of 
an associated elementary stream; "EP_stream_type[0] 
- [Ne-1]" indicating the type of the EP_map_for_one_ 
stream; "number_of_EP_High_entries[0] - [Ne - 1]" indi- 
cating the number of EP_High entries in the EP_map_ 
for_one_stream; "number_of_EP_Low_entries[0] - [Ne - 
1]" indicating the number of EP_Low entries in the EP_ 
map_for_one_stream; and "EP_map_for_one_stream_ 
PID_start_address[0] - [Ne - 1]" indicating the current 
address of the EP_map_for_one_stream. 
<Clip information: 2. EP_map> 

Hereinafter, the EP_map is described using a specific 
example. Fig. 9 illustrates the setting of an EP_map for 
a video stream carrying a movie (application_type = 1). 
The first level illustrates a plurality of pictures aligned in 
the presentation order. The second level illustrates the 
timeline of the pictures. The fourth level illustrates TS 
packets on the BD-ROM. The third level illustrates the 
EP_map setting. 

[0035] In this example. Access Units (Non-1 DR I pic- 



tures and IDR pictures) are located at points t1-t7 on the 
timeline illustrated on the second level. Provided that the 
time intervals between the points t1-t7 are about 1 sec, 
the EP_map of the movie video stream Is set so as to 
5 indicate each of the points t1-t7 with a corresponding 
entry time (PTS_EP_start) and a corresponding entry 
point (SPN_EP_start). 

[0036] Fig. 1 0 shows, in tabular form, pairs of EP_Low 
and EP_High values representing the PTS_EP_startand 

10 the SPN_EP_start of entry points #1 - #7 illustrated in Fig. 
9. In Fig. 10, the left table shows the values of EP_Low 
and the right table shows the values of EP_High. 
In Fig. 10, of the EP_Low(0) - (Nf - 1) shown in the left 
table, each PTS_EP_Low value of EP_Low(i) - (i + 3) 

15 indicate the least significant bits of the points t1 -t4. Also, 
each SPN_EP_Low value of the EP_Low(i) - (i + 3) indi- 
cate the least significant bits of the points n1 -n4. 
[0037] The right table in Fig. 10 shows the values of 
EP_High (0) - (Nc - 1 ) set in the EP_map. Provided that 

20 the points t1 -t4 have the same set of most significant bits 
and the points nl-n4 have the same set of most significant 
bits, the values of PTS_EP_High and SPN_EP_High are 
set to the respective sets of most significant bits. In ad- 
dition, the ref_to_EP_Low_id field corresponding to the 

25 EP_High is set, so that the first EP_Low field (EP_Low 
(i)) out of the EP_Low fields indicating the points t1-t4 
and n1-n4. With this setting, the EP_High indicates the 
common most significant bits of the PTS_EP_start and 
the common most significant bits of the SPN_EP_start. 

30 This concludes the description of the Main Clip and Clip 
information stored on the BD-ROM. Note that the 
00001 .mpis file is a file storing the PlayList information, 
but no description thereof is given here. It is because an 
identical file is stored on the local storage 200 and a de- 

35 scription of the PlayList information stored on the local 
storage 200 will be given later. 

[0038] Since the BD-ROM is a large-capacity record- 
ing medium, it is understood that the BD-ROM is used 
to supply video to the home theater system illustrated in 
40 Fig. 1 . This concludes the description of the BD-ROM. 
<Local Storage 200> 

Next, the following describes the local storage 200, which 
is a recording medium according to the present invention. 
Fig. 11 illustrates the internal structure of the local storage 

45 200. As illustrated in the figure, the recording medium 
according to the present invention can be manufactured 
by making improvements to the application layer. 
[0039] In the figure, the local storage 200 is illustrated 
on the fourth level, and the track of the local storage 200 

50 is illustrated on the third level. In the figure, the track is 
laterally stretched out, although the track in practice spi- 
rals outwards from the center of the local storage 200. 
The track consists of a lead-in area, a volume area, and 
a lead-out area. The volume area in the figure has a lay- 

55 ered structure made up of a physical layer, a file system 
layer, and an application layer. The first level of the figure 
illustrates the application format of the local storage 200 
using a directory structure. 
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[0040] In the directory structure illustrated in the figure, 
the local storage 200 has a ROOT directory. The ROOT 
directory has a subdirectory called "organization #1 and 
the "organization #1 " has a subdirectory called "disc #1 ". 
The "organization #1 " directory is assigned to a specific 
movie provider, whereas the "disc #1" directory is as- 
signed to a specific BD-ROM supplied from that provider. 
[0041] Since a specific directory is assigned to a spe- 
cific provider and subdirectories are provided within the 
directory in one-to-one correspondence with BD-ROMs 
supplied from that provider, download data relating to 
each BD-ROM is stored separately in the respective sub- 
directories. Similarly to the contents stored on the BD- 
ROM, the subdirectories contain PlayList information 
(00002. mpis), Clip information (00002.clpi, 00003. dpi, 

00004. clpi, and OOOOS.clpi), and AV clips (00002. m2ts, 
00003. m2ts, 00004.m2ts, and 00002. m2ts). 

[0042] Next, the following describes PlayList informa- 
tion, Clip information, and AV clips, all of which are com- 
ponent of the local storage 200. 
<Structure of Local Storage 200: 1 . AV Clip> 
The AV clips (00002. m2ts, 00003. m2ts, 00004. m2ts, and 

00005. m2ts) stored on the local storage 200 constitutes 
SubClips each of which is an AV clip composed of one 
ore more Out-of-MUX streams. An Out-of-MUX stream 
refers to an elementary stream to be presented during 
playback of an AV clip containing a video stream but not 
multiplexed with the video stream. In addition, a process 
of reading and decoding an Out-of-MUX stream to play 
back the Out-of-MUX stream during video stream play- 
back is referred to as "Out-of-MUX stream frame work". 
[0043] The types of Out-of-MUX streams include "pri- 
mary audio stream", "secondary audio stream", "presen- 
tation graphics (PG) stream", and "interactive graphics 
(IG) stream". 

According to the embodiment 1, among four AV clips 
illustrated in Fig. 11, the 00002.m2ts file contains a pri- 
mary audio stream, the 00003. m2ts file contains a sec- 
ondary audio stream, the 00004. m2ts file contains a PG 
stream, and the 00005 .m2ts file contains an IG stream. 
Yet, the stored contents are just one example and it is 
applicable that the four Out-of-MUX streams are multi- 
plexed into one SubClip. Hereinafter, the details of Out- 
of-MUX streams will be described. 
<Description of Out-of-MUX Stream: 1. Primary Stream 
& Secondary Stream> 

The "primary audio stream" carries main audio data, 
whereas the "secondary audio stream" caries auxiliary 
audio data. At the time of SubClip playback, audio play- 
back of the secondary audio stream is mixed with audio 
playback of the primary audio stream before output. Ex- 
amples of audio data carried by the secondary audio 
stream include "commentary". In the case where the pri- 
mary audio stream provides speech and BGM of a movie, 
which is a main feature, and the secondary stream pro- 
vides director's commentary, the sound of speech and 
BGM of the movie is mixed with the audio of the com- 
mentary before output. 



[0044] The secondary audio stream is stored only on 
the local storage 200 and supplied for playback, but not 
stored on the BD-ROM. On the other hand, the primary 
audio stream may be stored on the BD-ROM or on the 

5 local storage 200. In addition, the primary audio stream 
may be coded using a different scheme from that used 
for coding the secondary audio stream. 
[0045] Fig. 1 2 illustrates the internal structure of either 
of the primary and secondary audio streams. In the figure, 

10 the first level illustrates the timeline referenced during 
SubClip playback (SubClip timeline). The second level 
illustrates TS packets constituting the SubClip. Thethird 
level illustrates PES packets constituting the SubClip. 
The fourth level illustrates audio frames constituting the 

15 primary or secondary audio stream. As illustrated in the 
figure, the SubClip is manufactured by converting frames 
of the primary or secondary audio stream into PES pack- 
ets (illustrated on the third level), and further converting 
the PES packets into TS packets (illustrated on the sec- 

20 ond level). 

[0046] The header of a PES packet contains PTS in- 
dicating the presentation start time of the audio frame 
contained in the PES packet. That is, with reference to 
the PTS, the playback timing of the audio frame contained 

25 in the PES packet is specified on the SubClip timeline. 
Thus, the header of each PES packet is subjected to the 
stream analysis. 

<Description of Out-of-MUX Stream: 2. PG Stream> 
A PG stream is an elementary stream for synchronously 

30 presenting subtitles with video playback. Fig. 13 illus- 
trates the internal structure of a PG stream. In the figure, 
the fourth level illustrates PES packets constituting the 
SubClip, and the third level illustrates TS packets con- 
stituting the SubClip. The second level illustrates the 

35 SubClip timeline. The first level illustrates composite pic- 
tures obtained by decoding and overlaying the PG stream 
contained in the SubClip on the video stream contained 
in the Main Clip. 

[0047] The following describes the structure of the PES 
"^0 packets constituting the SubClip, illustrated on the fourth 
level. 

Each PES packet of the SubClip is generated by attach- 
ing a PES packet header to a sequence of functional 
segments, namely PCS (Presentation Control Segment), 

45 PDS (Pallet Definition Segment), WDS (Window Defini- 
tion Segment), ODS (Object Definition Segment), and 
END (END of Display Set Segment). 
[0048] The ODS defines graphics data for presentation 
of text subtitles. 

50 The WDS defines a rendering area for the display of 
graphics data on a screen. 

The PDS defines output colors for rendering the graphics 
data. 

[0049] The PCS defines page control for presentation 
55 of text subtitles. The page control includes Cut-In/Out, 
Fade-In/Out, Color Change, Scroll, Wipe-In/Out. Under 
the page control defined by PCS, it is possible to realize 
a display effect of gradually erasing a set of text subtitles 
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and presenting a next set of text subtitles. 
The END defines the end of the sequence of functional 
segments defining presentation of text subtitles. 
[0050] The PES packet header includes timestamps, 
such as PTS and DTS. The timestamps indicate the start 
timing of decoding the functional segments and the pres- 
entation timing of the graphics defined by the functional 
segments. The sequence of functional segments starting 
with the PCS and end with END is referred to as a "Display 
Set (DS)". The third level illustrates TS packets obtained 
by converting the PES packets. The second level illus- 
trates the timeline referenced forplaybackof the SubClip 
(SubClip timeline). The DTS included in the PCS indi- 
cates on the SubClip timeline the timing for decoding the 
PCS. The PTS included in the PCS indicate on the Sub- 
Clip timeline the presentation timing of graphics defined 
by the DS, which starts with the PCS. At the timing indi- 
cated by the PTS, the composite pictures illustrated on 
the first level are displayed. 

[0051] The types of DSs include "Epoch Start", "Ac- 
quisition Point", "Normal Case", and "Epoch Continue". 
An "Epoch Start" DS indicates the start of a new Epoch. 
An Epoch is a time period of continuous memory man- 
agement on the timeline of the AVclip playback, and also 
refers to a group of data allocated to that time period. 
Thus, an Epoch Start DS contains all functional segments 
necessary for the next screen composition. An Epoch 
Start DS is provided at a point that will be accessed during 
skip playback, such as the start of a chapter in the movie. 
[0052] An "Acquisition Point" DS is not the start of an 
Epoch but contains all functional segments necessary 
for the next screen composition. Therefore, graphics can 
be reliably displayed if playback is started from an Ac- 
quisition Point DS. That is to say, an Acquisition Point 
DS enables a display composition to be made from a 
midpoint in the Epoch. An Acquisition Point DS is provid- 
ed at a point to which a skip operation may be made. 
[0053] A "Normal Case" DS contains only a difference 
from the immediatelypreceding DS. Forexample, if the 
subtitles presented by a DSv are identical to the subtitles 
presented by the immediately preceding DSu but differ- 
ent in screen layout, the DSv contains a PCS and an 
END only and serves as a Normal Case DS. This ar- 
rangement eliminates the need to provide overlapping 
ODSs in DSs, so that the amount of data stored on the 
BD-ROM is reduced. Since a Normal Case DS only con- 
tains the difference, no display composition can be pre- 
sented with the Normal Case DS alone. 
[0054] An "Epoch Continue" DS indicates the Epoch 
of the AV clip continues from another AV clip if playback 
of the two AV clips shall be continuous. This concludes 
the description of the functional segments constituting 
the PG stream. 

<Description of Out-of-MUX Stream: 3. IG Stream> 
The IG stream is an elementary stream for synchronous 
presentation of text subtitles with video playback. 
[0055] Fig. 1 4 illustrates the internal structure of an IG 
stream. The fourth level illustrates PES packets consti- 



tuting a SubClip. The third level illustrates TS packets 
constituting the SubClip. The second level indicates the 
SubClip timeline. The first level illustrates composite pic- 
tures obtained by decoding and overlaying the IG stream, 

5 which is an Out-of-MUX stream, and the video stream, 
which is the Main Clip. 

Now, the following describes the PES packets illustrated 
on the fourth level. 

[0056] The PES packets of the SubClip is generated 

'io by attaching a PES packet header to a sequence of func- 
tional segments, namely ICS (Interactive Composition 
Segment), PDS (Pallet Definition Segment), ODS (Ob- 
ject Definition Segment), and END (END of Display Set 
Segment). 

^5 The ODS defines graphics data used for rendering a but- 
ton. 

[0057] The PDS defines output colors used for render- 
ing graphics data. 

The ICS defines interactive control for changing the but- 

20 ton state in response to a user operation. 

The END defines the end of the sequence of functional 
segments defining presentation of a menu. 
[0058] The header of a PES packet includes times- 
tamps, such as PTS and DTS. The timestamps indicate 

25 the start timing of decoding the functional segments and 
the presentation timing of the graphics defined by the 
functional segments. The sequence of functional seg- 
ments starting with the PCS and end with EN D is referred 
to as a "Display Set". Similarly to DSs contained in the 

30 PG stream, the types of DSs include "Epoch Start", "Ac- 
quisition Point", "Normal Case", and "Epoch Continue". 
[0059] In Fig. 14, the third level illustrates TS packets 
obtained by converting the PES packets. The second 
level illustrates the time line referenced during SubClip 

35 playback (SubClip timeline). The DTS included in the ICS 
indicates on the SubClip timeline the timing for decoding 
the ICS. The PTS included in the ICS indicate on the 
SubClip timeline the presentation timing of graphics de- 
fined by the DS, which starts with the ICS. At the timing 

"^0 indicated by the PTS, the composite pictures illustrated 
on the first level are displayed. 

[0060] A sequence of functional segments including 
ICS, PDS, ODS, and END is referred to as a "Display 
Set", which is a group of functional segments for realizing 

45 display of one menu. 

This concludes the description of SubClip. 
<Structure of Local Storage 200: 2. Clip lnformation> 
The following describes Clip information stored on the 
local storage 200. Fig. 1 5 illustrates the data structure of 

50 Clip information stored on the local storage 200. The Clip 
information stored on the local storage 200 is identical in 
data structure to the Clip information stored on the BD- 
ROM 200. Yet, the application_type, EP_map, and EP_ 
stream_type fields are set to the values specific to the 

55 SubClip. 

<Description of Clip Information in SubClip: 1. 

application_type> 

The following describes the application_type field illus- 
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trated in Fig. 15. If tine SubClip is one of primary audio 
stream, secondary audio stream, PG stream, and IG 
stream mentioned above, the application_type is set to 
tine value "7". 

[0061 ] The application_type = 7 indicates that the Sub- 
Clip multiplexed with the Clip information is "additional 
contents without video". Since no video is contained, the 
SubClip is designed to be stored to the local storage 200 
via the network and to be supplied to a playback device 
from the local storage 200 rather than from the BD-ROM. 
This concludes the description of the setting of the 
application_type field in the SubClip. 
[0062] <Clip Information in SubClip: 1. Out-of-MUX_ 
EP_map> 

Next, the EP_map provided for the SubClip is described. 
The EP_map fields on the local storage 200 are present- 
ed one for each Out-of-MUX stream multiplexed in the 
SubClip. Each EP_map indicates a plurality of entry 
points of the associated Out-of-MUX stream, along with 
a plurality of entry times corresponding to the entry points. 
[0063] The types of Out-of-MUX streams include "pri- 
mary audio stream", "secondary audio stream", "PG 
stream", and "IG stream". Each type of Out-of-MUX 
stream differs in point on the stream at which the stream 
becomes available for playback. More specifically, the 
primary and secondary audio streams are each com- 
posed of a plurality of audio frames and playback can 
basically be started from the head of any audio frame. 
[0064] In the case of PG and IG streams, on the other 
hand, it is required that a Display Set composed of a 
complete set of functional segments be supplied to a de- 
coder. A DS composed of a complete set of functional 
segments refers to a DS other than "Normal Case". That 
is, "Epoch Start", "Acquisition Point", and "Epoch Con- 
tinue" DSs are each composed of a complete set of func- 
tional segments, and the first PCS and the first ICS in 
such DSs serve as entry points. 

[0065] As described above, each type of Out-of-MUX 
stream differs in point on the stream at which decoding 

of the stream can be started. Thus, each EP_map has a 
different structure depending on the type of Out-of-MUX 
stream associated with the EP_map. An EP_map asso- 
ciated with an Out-of-MUX stream is referred to as an 

"Out-of-MUX_EP_map". 

<Details of Out-of-MUX_EP_map: 2. Out-of-MUX_EP_ 
map for Audio 

Next, the following describes the structure of EP_map 
associated with a primary or secondary audio stream. 
The EP_map associated with a primary or secondary au- 
dio stream differs from the EP_map associated with video 
in time intervals between adjacent entry points. More 
specifically, in the EP_map for video, entry points are set 
at relatively short time intervals not greater than one sec- 
ond. On the other hand, entry points in the EP_map for 
audio are set at longer time intervals of five seconds. 
[0066] Fig. 16 illustrates the EP map for a primary or 
secondary audio stream. The structure of the EP_map 
illustrated in the figure is such that entry points are set 



so as to correspond to entry times set at constant time 
intervals of five seconds. In the figure, the third level il- 
lustrates the SubClip timeline, and t1-t6 on the SubClip 
timeline denote entry times. The entry times t1 , t2, t3, t4, 
5 t5, and t6 are set at constant time intervals of five sec- 
onds. The second level illustrates the EP_map. Each 
PTS_EP_start field illustrated on the second level indi- 
cates an entry time. 

[0067] The first level illustrates TS packets constituting 

10 the primary or secondary audio stream. Among the TS 
packets, SPN = n1, n2, nS, n4, n5, and n6 are set to 
indicate entry points, and the SPN_EP_start fields illus- 
trated on the second level are set to indicate n 1 -n6. Since 
the SPN_EP_start of each entry point is set to correspond 

15 PTS_EP_start, the entry times at five seconds intervals 
correspond to entry points. 

[0068] In the EP_map of the primary or secondary au- 
dio stream, the interval between each two adjacent entry 
points may be the data interval of 256K bytes. This inter- 
ne val of 256K bytes is calculated by multiplying the transfer 
rate of the secondary audio stream by the time interval 
of five seconds. Since the time interval set in the EP_ 
map is five seconds, the stream analysis needs to cover 
at most five seconds of data. This concludes the descrip- 
25 tionofthe EP_map set for the primary or secondary audio 
stream. 

<Details of Out-of-MUX_EP_map: 3. EP_map for PG 
Stream> 

Next, an EP_map set for a PG stream will be described. 

30 [0069] Fig. 17 illustrates the EP_map set on the PG 
stream timeline. In the figure, the first level illustrates TS 
packets constituting the PG stream. The second level 
illustrates the EP_map. The third level illustrates the Sub- 
Clip timeline. On the first level, the first PCS in each DS 

35 which is not a Normal Case DS is present at points spec- 
ified by SPN = n1 and n5. These points n1 and n5 are 
determined as entry points. On the other hand, the PTSs 
of the PCSs specify the temporal points t1 and t5 on the 
SubClip timeline illustrated on the third level. Thus, the 

"^0 EP_map shows SPN = n1 and n5, together with the cor- 
responding PTS = t1 and t5, respectively. 
[0070] As above, the head of each complete set of 
functional segments coincides with an entry time shown 
by the PTS of PCS. Consequently, with reference to the 

45 EP_map, presentation of graphics can be started from 
some mid point of the stream without conducting stream 
analysis. 

<Details of Out-of-MUX_EP_map: 4. EP_map for IG 
Stream> 

50 Fig. 1 8 illustrates the EP_map set on the IG stream time- 
line. In the figure, the first level illustrates TS packets 
constituting the IG stream. The second level illustrates 
the EP_map. The third level illustrates the SubClip time- 
line. On the first level, the first ICS in each DS which is 

55 not a Normal Case DS is present at points specified by 
SPN = n1 and n5. These points n1 and n5 are determined 
as entry points. On the other hand, the PTSs of the ICSs 
specify the temporal points t1 and t5 on the SubClip time- 
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line illustrated on the third level, the EP_map show the 
SPN = n1 and n5, together with the corresponding PTS 
= t1 and t5, respectively. 

[0071] As above, the head of each complete set of 
functional segments coincides with an entry time shown 5 
by the PTS of ICS. Consequently, with reference to the 
EP_map, presentation of menu display can be started 
from some mid point of the stream without conducting 
stream analysis. 

The above description clarifies that the structure of EP_ 
map differs depending on the type of Out-of-MUX stream 
associated with the EP_map. This concludes the descrip- 
tion of the EP map included in the Clip information stored 
on the local storage 200. The following describes the EP_ 
stream_type field included in the Clip information stored ^5 
on the local storage 200. 

<Description of Clip Information in SubClip: 3. EP_ 
stream_type> 

20 

[0072] The EP_stream_type is provided for each Out- 
of-MUX stream multiplexed in the SubClip and indicates 
one of the EP_map structures illustrated in Figs. 16-18. 
That is, when i-th Out-of-MUX stream multiplexed in one 
SubClip is denoted as Out-of-MUX stream[i], the EP_ 25 
stream_type[i] indicates the type of EP_map_for_one_ 
streamassociatedwith the Out-of-MUX stream[i] and set 
to one of the values of "3", "4", "6", and "7". When set to 
the value of "3" or "4", the EP_stream_type indicates the 
EP map structure specific to a primary or secondary au- so 
dio stream. That is, the EP_map of that type shows entry 
times set at constant time intervals or entry points set at 
constant data intervals, as illustrated in Fig. 16. When 
set to the value of "6", the EP_stream type indicates the 
EP_map structure specific to a PG stream. That is, the 35 
EP_map of that type shows, as entry points, the head of 
each DS other than Normal Case DS (as illustrated in 
Fig. 17). When set to the value of "7", the EP_stream 
type indicates the EP_map structure specific to the IG 
stream. That is, the EP_map of that type shows, as entry ^0 
points, the head of each DS other than Normal Case DS 
(as illustrated in Fig. 18). 

[0073] Since the EP_stream_type clearly indicates the 

structure of the associated EP_map, it is known whether 
stream analysis is required from an entry point shown by ^5 
the EP_map or data is to be read starting from an entry 
point shown by the EP_map and supplied to the decoder. 
<Description of Clip Information in SubClip: 4. Signifi- 
cance of EP_stream_type> 

The following describes the technical significance of the 50 
provision of the EP_stream_type field in the Out-of- 
MUX_EP_map field. 

[0074] In the case where an entry time is specified as 
an access point, playback can be started immediately. 
However, in the case where a requested access point 55 
does not coincide with an entry time, stream analysis 
needs to be carried out. If no information is provided as 
to the time intervals between adj acent entry times or the 



types of data designated as entry points, the playback 
device cannot estimate the maximum degrees of stream 
analysis that would be sufficient. Not knowing the maxi- 
mum degree of stream analysis that would be required, 
it is possible that a response of the playback device to a 
user operation will be notably delayed. 
[0075] In view of the above undesirable possibility, the 
Out-of-MUX_EP_map includes the EP_stream_typethat 
indicates the time intervals between entry times and the 
data types of entry points. When set to the value of "3" 
or "4", the EP_stream_type indicates that the entry points 
are at constant time intervals of 5 sec. With reference to 
such EP stream type, the playback device will recog- 
nize that stream analysis, if required, covering 5 seconds 
of data would be sufficient. In other words, the playback 
device will be informed that any desired point can be 
accessed at random by analyzing at most five seconds 
data of the Out-of-MUX stream. 

[0076] When set to either "6" or "7", the EP_stream_ 
type indicates that the entry points are located at the 
heads of DSs each composed of a complete set of func- 
tional segments. When starting playback, the playback 
device reads functional segments starting from any of 
the entry points located at in constant intervals. Conse- 
quently, it is ensured that subtitles or a menu is presented 
starting from any intended playback point without stream 
analysis. 

[0077] Since the EP_stream_type causes the play- 
backdevice to judge whether up to five seconds of stream 
data needs to be analyzed or no stream analysis is re- 
quired at all. Thus, even if not only the Main Clip but also 
the SubClip needs to be accessed at random, the play- 
back device is not subject to excessive burden. As a re- 
sult of the reduced burden, response of the playback de- 
vice to a user operation improves. 
[0078] This concludes the description of the Clip infor- 
mation stored on the local storage 200. 
<Structure of Local Storage 200: 3. Play List lnformation> 
Next, a description is given of PlayList information stored 
on the local storage 200. Each file with the extension 
"mpis" (00001. mpis) stores PlayList (PL) information. 
PlayList information defines, as a PlayList (PL), two play- 
back paths called "Main Path" and "SubPath" bundled 
together. Fig. 1 9 illustrates the data structure of PlayList 
information. As illustrated in the figure, the PlayList infor- 
mation is composed of: MainPath information (MainPath 
0) definingaMainPath; PlayListMark information (PlayL- 
istMarkO) defining chapters; and SubPath information 
(SubPathQ) defining a SubPath. 

[0079] A Main Path is a playback path defined on the 
main AV clip, whereas the SubPath is a playback path 
defined on the SubClip. 

<PlayList Information: 1. MainPath lnformation> 
First, the Main Path is described. The main path is a 
playback path defined on a video or audio stream carrying 
a main feature. 

[0080] As indicated by arrows mp1, the Main Path in- 
cludes a plurality of pieces of Playltem information (... 
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Playltem ( )...). Each piece of Playltem information de- 
fines one or more logical units eacli specify a playback 
section constituting the Main Path. As leader lines hsi 
indicate, the structure of a piece of Playltem Information 
is illustrated in detail. As illustrated, the Playltem infor- 
mation is composed of the following fields: "Clip_ 
information_file_name" indicating the name of a file con- 
taining the playback section information of the AV clip to 
which the In-point and Out-point of the playback section 
belong; "Clip_codec_identifier" indicating the coding type 
of the AV clip; and "ln_time" which is timing information 
indicating the start point of the playback section; and 
"Out_time" which is timing information indicating the end 
point of the playback section. 

[0081] Fig. 20 illustrates the relationship between the 
Main Clip and the PlayList information. The first level il- 
lustrates the timeline of the PlayList information. The sec- 
ond to fifth levels illustrate the video stream referenced 
to by the EP_map (identical to Fig. 5). 
The PlayList information includes two pieces of Playltem 
information, which are Playltem lnfo#1 and Playltem Info 
#2. The pairs of ln_time and Out_time in the respective 
pieces of Playltem Info #1 and #2 define two playback 
sections. When aligning the two playback sections, a 
timeline that is different from the AV clip timeline is de- 
fined. This timeline is the Playltemtimeline illustrated on 
the first level. As described herein, by defining Playltem 
information, a different timeline from the AV clip timeline 
is defined. 

<Description of PlayList Information: 2. PlayListMark> 

[0082] This concludes the description of Playltem in- 
formation according to the embodiment 1 . Now, PlayL- 
istMark information will be described. 
Fig. 21 illustrates the internal structure of PlayListMark 
information included in the PlayList information. As indi- 
cated by leader lines pmO, the PlayListMark information 
is composed of a plurality of pieces of PLMark information 
(PLMark #1 to PLMark #n). Each piece of PLMark infor- 
mation (PLMarkQ) specifies an arbitrary portion on the 
PL timeline as a chapter point. As indicated by leader 
lines pml, the PLMark information is composed of the 
following fields: "ref_to_Playltem_id" indicating a Play- 
ltem in which a chapter is to be designated; and "mark_ 
time_stamp" specifying the position of a chapter in the 
Playltem using time notation. 

[0083] Fig. 22 illustrates chapter positions specified by 
the PLMark information included in the PlayList informa- 
tion. The second to fifth levels illustrate the EP_map and 
the AV clip identical to those illustrated in Fig. 20. 
The first level in the figure illustrates the PlayListMark 
information and the PL timeline. On the PL timeline, two 
pieces of PLMark information #1 and #2 are present. Ar- 
rows ktl and kt2 in the figure represent the designation 
by the ref to Playltem id. As shown by the arrows, the 
ref_to_Playltem_id in the respective pieces of PLMark 
information designate the respective pieces of Playltem 



information. In addition, each Mark_time_stamp indi- 
cates a portion on the Playltem timeline to be designated 
as Chapter #1 and Chapter #2. As described herein, 
PLMark information defines chapter points on the Play- 
5 Item timeline. 

<Description of PlayList Information: 3. SubPath Infor- 
mation> 

As described above, the Main Path is a playback path 
defined within the Main Clip containing main feature vid- 

10 eo. 

On the other hand, the SubPath is a playback path de- 
fined within a SubClip to be played back in synchronism 
with the Main Path. 

[0084] Fig. 23 illustrates the internal structure of Sub- 
15 Path information in detail. As arrows hcO in the figure 
indicate, each SubPath information includes the follow- 
ing fields: "SubPath_type" indicating the type of the Sub- 
Clip; and at least one piece of SubPlayltem information 
(... SubPlayltem( ) ...). 
20 in the figure, leader lines hcl indicate that the structure 
of SubPlayltem information is illustrated in detail. As in- 
dicated by arrows hc1, the SubPlayltem information is 
composed of the following fields: "Clip_information_file_ 
name", "SubPlayltem_ln_Time", "SubPlayltem_Out_ 
25 Time", "sync_Playltem_id", and "sync_start_PTS_of_ 
Playltem". 

[0085] The Clip_information_file_name specifies the 
name of file containing Clip information, so that the Sub- 
Clip associated with the SubPlayltem is uniquely identi- 
30 fled. 

The SubPlayltem_ln_time indicates on the SubClip time- 
line the start point of the SubPlayltem. 
The SubPlayltem_Out_time indicates on the SubClip 
timeline the end point of the SubPlayltem. 

35 [0086] The sync_Playltem_id uniquely specifies one 
of the Playltems constituting the Main Path, to be syn- 
chronized with the SubPlayltem. The SubPlayltem_ln_ 
time is present on the playback timeline of the Playltem 
specified by the sync Playltem id. 

"^0 The sync start_PTS_of_Playltem specifies a point cor- 
responding, on the playback timeline of the Playltem 
specified by the sync_Playltem_id, to the start point of 
the SubPlayltem specified by the SubPlayltem_ln_time. 
<SubPath Information: 1. SubPath_type> 

45 This concludes the description of SubPath information. 
Next, the SubPath_type is described. The SubPath_type 
is set to a value from 0 to 255 to indicate the type of the 
SubPath specified by the SubPath information. 
[0087] The value of the SubPath_type is set in relation 

50 to the contents of the Clip information specified by the 
Clip_information_file_name included in the SubPlayltem 
information, and more specifically to the application_type 
included in the Cliplnfo. For example, the SubPath_type 
is set to a value from 0 to 255, and when the application_ 

55 type of the Cliplnfo is set to "7", the only possible values 
of the SubPath type are from 5 to 8. 
With the SubPath_type set to "5", the SubPath informa- 
tion specifies a SubPath that is a playback path on the 
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primary audio. Tine playback patli on the primary audio 
is defined for appending or replacing the primary audio 
playback defined by the Playltem information. 
[0088] With the SubPath_type set to "6", the SubPath 
information specifies a playback path on Presentation 5 
Graphics. The playback path on the PG stream according 
to the SubPath information is defined for appending or 
replacing the PG streamdeflnedby the Playltem Informa- 
tion. 

With the SubPath_type set to "7", the SubPath informa- io 
tion specifies a playback path on Interactive Graphics. 
The playback path on the IG stream Is defined for ap- 
pending or replacing the PG stream defined by the Play- 
ltem information. 

[0089] With the SubPath_type set to "8", the SubPath ^5 
Information specifies a playback path on secondary au- 
dio. The playback path on the secondary audio stream 

is defined for audio mixing with the Primary audio defined 
by the Playltem information. 

This concludes the description of the SubPath_type. 20 
<Detalls of SubPath Information: 2. Relationship among 
Three> 

The three used herein refers to the SubClip, the PlayList 
information, and the Main Clip. The SubClip and the Play- 
List Information are both stored on the local storage 200, 25 
whereas the Main Clip is stored on the BD-ROM. 
[0090] Fig. 24 illustrates the relationship among the 
Main Clip stored on the BD-ROM and the SubClip and 
the PlayList Information both stored on the local storage 
200. In the figure, the first level Illustrates the SubCllps 30 
stored on the local storage 200. As illustrated, the Sub- 
Clips stored on the local storage 200 include the primary 
audio stream, secondary audio stream, PG stream, and 
IG stream. One of the SubStreams specified as the Sub- 
Path is supplied for synchronous playback with the Main 35 
Stream. 

[0091] The second level illustrates the two timelines 
defined by the PlayList Information. The lower one Is the 
Playltem timeline defined by the Playltem Information 
and the upper one is the SubPlayltem timeline defined 
by the SubPlayltem. 

As Illustrated In the figure, the SubPlayltem_Cllp_ 
lnformatlon_flle_name specifies one of the fourSubCllps 
illustrated on the first level and a portion of the specified 
SubClip is designated as a playback section. ^5 
[0092] The SubPlayltem_ln_tlme and the 
SubPlayltem_Out_time define the start and end points 
of the playback section of the SubClip. 
The sync_Playltem_id represented by an arrow in the 
figure specifies a Playltem to be synchronized with the 50 
SubClip. The sync_start_PTS_of_Playltem Indicates the 
difference between the origin points of the Playltem time- 
line and the SubPlayltem timeline. 
[0093] Fig. 25 illustrates the EP_map for the Main Clip, 
the Playltem timeline, the EP_map for the SubClip, which 55 
Is either a primary or secondary audio stream, and the 
SubPlayltem timeline. 

In the figure, the middle level and the fourth to first levels 



from the bottom illustrate what is illustrated in Fig. 20. 
More specifically, the Playltem timeline, the sequence of 
pictures, the MalnClip timeline, the EP_map, and the se- 
quence of TS packets are Illustrated In the stated order. 
[0094] The first to third levels from the top illustrate 
what is illustrated in Fig. 16. More specifically, the TS 
packets, the EP_map, the SubClip timeline are illustrated 
In the stated order. The fourth level from the top Illustrates 
the SubPlayltem timeline Illustrated In Fig. 24. For the 
Main Clip, the entry times are set at time intervals of 1 
second. For the SubClip, the entry times are set at time 
Intervals of five seconds. 

Fig. 26 Illustrates the EP mapforthe Main Clip, the Play- 
ltem timeline, the EP_mapforthe SubClip, which is either 
a PG or IG stream. 

[0095] In the figure, the fourth to first levels from the 
bottom Illustrate what Is Illustrated In Fig. 20. More spe- 
cifically, the Playltem timeline, the sequence of pictures, 
the MainClip timeline, the EP_map, and the sequence of 
TS packets are Illustrated in the stated order. 
The first and third levels from the top Illustrate what Is 
Illustrated in Fig. 16. More specifically, the sequence of 
TS packets, the EP_map, and the SubClip timeline are 
Illustrated in the stated order. The fourth level from the 
top Illustrates the SubPlayltem timeline Illustrated In Fig. 
24. For the Main Clip, the entry points are set at the time 
intervals of one second. For the SubClip, the positions 
of DSs other than a Normal Case DS are set as entry 
points. 

[0096] <Detalls of SubPath Information: 3. Interrela- 

tionship> 

Fig. 27 shows a table of permissible combinations of the 
values of the SubPath_type, the appllcation_type, and 
the EP_stream_type. 

As shown in the table, when the SubPath_type is set to 
"5" and the application_type is set to "7", the EP_stream_ 
type Is set to "3". Since the EP_stream_type Is set to "3", 
the playback device will recognize that the EP_map of 
this EP_stream_type is provided for primary audio play- 
back and thus the entry points are set at time intervals 
of five seconds or data intervals of 256K bytes. 
[0097] In the case where the SubPathJype Is set to 
"6" and the appllcatlon_type Is set to "7", the EP_stream_ 
type is set to "6". Since the EP_stream_type is set to "6", 
the playback device will recognize that the EP_map of 
this EP_stream_type Is provided for playback of Presen- 
tation Graphics and thus the entry points are set at the 
heads of Display Sets each composed of a complete set 
of functional segments. 

[0098] In the case where the SubPath_type Is set to 
"7" and the appllcatlon_type Is set to "7", the EP_stream_ 
type is set to "7". Since the EP_stream_type is set to "7", 
the playback device will recognize that the EP_map of 
this EP_stream_type is provided for playback of Interac- 
tive Graphics and thus the entry points are set at the 
heads of Display Sets each composed of a complete set 
of functional segments. 

[0099] In the case where the SubPath_type is set to 
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"8" and the application_type is set to "7", the EP_stream_ 
type is set to "4". Since the EP_stream_type is set to "4", 
the playback device will recognize that the EP_map of 
this EP_stream_type is provided for secondary audio 
playback and thus the entry points are set at time intervals 
of five seconds or data intervals of 256K bytes. 
[01 00] This concludes the description of PlayList infor- 
mation, which is a component of the local storage 200. 
This concludes the overall description of the local storage 
200. 

<Virtual File System> 

Now, a virtual file system will be described. Fig. 28 illus- 
trates the virtual file system created by a playback device 
300. The top left block of thefigure illustrates the contents 
stored on the BD-ROM. The bottom left block illustrates 
the contents stored on the local storage 200. The right 
block illustrates the structure of the virtual file system. 
[0101] The playback device creates the virtual file sys- 
tem through the combined use of the AV clip, the Clip 
information, the PlayList information stored on the BD- 
ROM, with the AV clip, the Clip information, the PlayList 
information stored on the local storage 200. 
More specifically, the virtual file system is created by: 

i) adding the PlayList (00002.MPLS) on the local 
storage into the MPLS directory on the BD-ROM; 

ii) adding the Clip Info #2, #3, #4, and #5 
(00002. CLPI, 00003.CLPI, 00004. CLPI, and 
00005.CLPI) on the local storage into the CLPI di- 
rectory on the BD-ROM; and 

iii) adding the AV clip #2, #3, #4, and #5 
(00002. M2TS, 00003.M2TS, 00004. M2TS, and 
00005.M2TS) on the local storage to the STREAM 
directory on the BD-ROM. 

[0102] As a result of the above processing, the virtual 
file system as illustrated in the right block of Fig. 28 is 
created. 

This concludes the overall description of the recording 

medium according to the present invention. Next, a de- 
scription is given of the playback device according to the 
present invention. 

<lnternal Structure of Playback Device> 
Fig. 29 illustrates the internal structure of the playback 
device. Based on the internal structure illustrated in the 
figure, playback devices consistent with the present in- 
vention are industrially manufactured. The playback de- 
vice of the present invention is roughly composed of two 
parts, one of which is a system LSI and the other is a 
drive device. By mounting those parts into a device cab- 
inet and onto a substrate, the playback device can be 
manufactured industrially. The system LSI is an integrat- 
ed circuit containing various processing units for imple- 
menting the functions of the playback device. The play- 
back device manufactured in the above manner is com- 
posed of a BD drive 1 , an arrival time clock counter 2, a 
source de-packetizer 3, a PID filter 4, a transport buffer 
5, a multiplexed buffer 6, a coded picture buffer 7, a video 



decoder 8, a decoded picture buffer 10, a video plane 
11 , a transport buffer 1 2, a coded data buffer 1 3, a stream 
graphics processor 1 4, an object buffer 1 5, a composition 
buffer 16, a composition controller 17, a presentation 

5 graphics plane 1 8, a GLUT unit 1 9, a transport buffer 20, 
a coded data buffer 21 , a stream graphics processor 22, 
an object buffer 23, a composition buffer 24, a composi- 
tion controller 25, an interactive graphics plane 26, a 
CLUTunit27, a compositor 28, a compositor 29, a switch 

10 30, a network device 31 , a switch 32, an arrival time clock 
counter 33, a source de-packetizer 34, a PID filter 35, a 
switch 36, a transport buffer 37, an elementary buffer 38, 
an audio decoder 39, a transport buffer 40, a buffer 41 , 
an audio decoder 42, a mixer 43, a scenario memory 44, 

15 a controller 45, a main converter 46, a sub-converter 47, 
a PL playback controller 48, and a PSR set 49. Note that 
the internal structure illustrated in the figure is a decoder 
model based on the MPEG T-STD model and cabala of 
downconverting. 

20 [0103] The BD drive 1 loads/ejects the BD-ROM and 
accesses the BD-ROM to sequentially reads Aligned 
Units each composed of 32 ES-TS packets. 
The arrival time clock counter 2 generates an arrival time 
clock using a 27 MHz crystal oscillator (27 MHz X-tal). 

25 The arrival time clock is a clock signal defining the time- 
line on which the ATS assigned to each TS packet Is 
based. 

[0104] Once an Aligned Unit composed of 32 EX-TS 
packets is read from the BD-ROM, the source de-pack- 
so etizer 3 removes the TP extra header from each of the 
32 ES-TS packets and outputs the TS packets without 
the headers to the PID filter 4. The output by the source 
de-packetizer 3 to the PID filter 4 is performed at the 
timing when the time measured by the arrival time clock 
35 counter 2 reaches the ATS shown by theTP_extra_head- 
er. Since the output to the PID filter 4 is carried out in 
accordance with the ATS, the TS packets are sequen- 
tially output to the PID filter 4 in accordance with the time 
measured by the arrival time clock counter 2, regardless 
^0 of the speed at which data is read from the BD-ROM, 
such as 1 x-speed or 2x-speed. 
[0105] The PID filter 4 judges, with reference to the 
PID attached to the TS packets, the type of stream to 
which the TS packets belong is a video stream, a PG 
45 stream, an IG stream, or a primary audio stream. Accord- 
ing to the judgment, the PID filter4 outputs the TS packets 
to one of the transport buffer 5, the transport buffer 12, 
the transport buffer 20, and transport buffer 37. 
The transport buffer (TB) 5 is a buffer for temporarily ster- 
eo ing TS packets output from the PID filter 4, if the TS pack- 
ets belong to a video stream. 

[0106] The multiplexed buffer (MB) 6 is a buffer for 
temporarily storing PES packets output from the trans- 
port buffer 5, in order to later output the video stream to 
55 the coded picture buffer 7. 

The coded picture buffer (CPB) 7 is a buffer for storing 
coded pictures (I pictures, B pictures, and P pictures). 
[01 07] The video decoder 8 decodes individual frames 
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contained in tine video elementary stream at every pre- 
determined decoding time (DTS) to obtain a plurality of 
frames and renders the resulting picture data on the de- 
coded picture buffer 1 0. 

The decoded picture buffer 10 is a buffer on which de- 
coded picture data is rendered. 

The video plane 1 1 is used for presenting uncompressed 
picture data. A plane is a memory area of the playback 
device for storing a frame of pixel value data. The video 
plane 1 1 stores picture data at the resolution of 1 920 x 
1080, and the picture data is composed of pixel values 
each expressed by 1 6-bit YUV values. 
[01 08] The transport buffer (TB) 1 2 is a buffer for tem- 
porarily storing TS output from the PID filter 4, if the TS 
packets belong to a PG stream. 

The coded data buffer (CDB) 13 temporarily stores PES 
packets constituting a PG stream. 
The stream graphics processor (SPG) 14 decodes an 
ODS to obtain uncompressed graphics data expressed 
by index colors, and renders the obtained graphics data 
as a graphics object on the object buffer 15. Decoding 
by the stream graphics processor 14 is instantaneous, 
and the stream graphics processor 1 4 temporarily stores 
the resulting graphics object. Although the decoding by 
the stream graphics processor 14 Is Instantaneous, 
transfer of the graphics object to the object buffer 1 5 takes 
some time. In the player model conforming to the BD- 
ROM specification, the transfer rate to the object buffer 
15 is 128 Mbps. The end time of transfer to the object 
buffer 15 Is shown by the PTSInthe END segment. Thus, 
until the time shown by the PTS is reached, the process- 
ing on the subsequent DS is delayed. The rendering of 
the graphics object obtained by decoding each ODS Is 
started at the time shown by the DTS associated with 
that ODS and end by the time shown by the PTS asso- 
ciated with that ODS. 

[01 09] The obj ect buffer 1 5 holds the graphics obj ect 
obtained by decoding an ODS by the stream graphics 
processor 14. The object buffer 15 needs to be twice or 
four times as large as the graphics plane 18 for the fol- 
lowing reason. For implementing scrolling, it is necessary 
to store a graphics object that is two or four times as large 
as the graphics plane 18. 

[01 10] The composition buffer 1 6 is memory for storing 
PCS and PDS. In the case where there are two DSs to 
be processed and where the active period of their PCS 
overlap, the composition buffer 16 stores the plurality of 

PCS to be processed. 

The composition controller 1 7 judges, each time the cur- 
rent playback point reaches a new DS, whether the 
Composltlon_state of the PCS contained In the new DS 
indicates Epoch Start, Acquisition Point, or Normal Case. 
If the Epoch Start is indicated, the composition controller 
1 7 transfers the PCS on the coded data buffer 13 to the 
composition buffer 1 6. 

[0111] The presentation graphic plane 18 Is a memory 

area as large as one full screen and stores uncom- 
pressed graphics data worth one screen. The presenta- 



tion graphic plane 1 8 stores uncompressed graphics data 
at the resolution of 1 920 x 1 080 and the uncompressed 
graphics data Is composed of pixel values each ex- 
pressed using an 8-blt Index colors. By converting the 
5 index colors with reference to a GLUT (Color Lookup Ta- 
ble), the uncompressed graphics data stored on the pres- 
entation graphics plane 18 is supplied for display. 
[01 12] The CLUT unit 1 9 converts the index colors of 
the uncompressed graphics data stored on the presen- 
ce tation graphic plane 18 to Y, CR, and Cb values. 

The transport buffer (TB) 20 is a buffer for temporarily 
storing TS packets belonging to an IG stream. 
The coded data buffer (CDB) 21 Is a buffer for temporarily 
storing PES packets constituting an IG stream. 
C5 [0113] The stream graphics processor (SPG) 22 de- 
codes an ODS and transfers uncompressed graphics da- 
ta obtained by the decoding to the object buffer 23. 
The object buffer 23 stores a plurality of uncompressed 
graphics objects decoded by the stream graphics proc- 
20 essor 22. The rectangle area of the obj ect buffer 23 to 
be occupied by each graphics object is identified with 
reference to the object idincluded in the associated 
ODS. Consequently, when a graphics object enters the 
object buffer 23 and the objectjd of the graphics object 
25 is the same as that of the graphics object currently on 
the object buffer 23, the area occupied by the current 
graphics object is overwritten with the new graphics ob- 
ject. 

[01 1 4] The composition buffer 24 is a buffer for storing 
30 Interactive composition transported by one or more IC- 

Ss. The lnteractive_composition stored on the composi- 
tion buffer 24 is supplied to the composition controller 25 
for decoding. 

The composition controller 25 judges, each time the cur- 

35 rent playback point reaches a new DS, whether the com- 
position state of the ICS contained in the new DS indi- 
cates Epoch Start, Acquisition Point, or Normal Case. If 
the Epoch Start Is indicated, the composition controller 
25 transfers the new interactive composition on the cod- 

^0 ed data buffer 21 to the composition buffer 24. 

[0115] Each time the ICS contained in an acquisition 
point DS is supplied to the coded data buffer 21 , the com- 
position controller 25 compares the page information of 
the newly supplied ICS with the page information of the 

45 interactive_composition already stored in the composi- 
tion buffer 24. If the page information with a greater value 
page_version_n umber is present on the coded data buff- 
er 21 , the page information is transferred to the compo- 
sition buffer 24. As a result, the intended page information 

50 on the composition buffer 24 is updated. The composition 
controller 25 further judges whether the thus updated 
page information is currently being displayed. If the up- 
dated page is being displayed, the page is rendered again 
to reflect the update. 

55 [0116] The Interactive graphics plane 26 is used for 
presenting uncompressed graphics data obtained by de- 
coding graphics object by the stream graphics processor 
(SGP) 22. The graphics data is rendered on the interac- 
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tive graphics plane 26 at the resolution of 1 920 x 1 080, 
and the graphics data is composed of pixel values each 
expressed using 8-bit index colors. By converting the in- 
dex colors with reference to the GLUT (Color Lookup 
Table), the uncompressed graphics data stored on the 5 
interactive graphics plane 26 is supplied for presentation. 
[01 1 7] The GLUT unit 27 converts the index colors of 
the uncompressed graphics data stored on the interac- 
tive graphics plane 26 to Y, CR, and Cb values. 
The compositor 28 overlays the uncompressed frame '^o 
data rendered on the video plane 1 1 with the uncom- 
pressed graphics object rendered on the presentation 
graphic plane 1 8. As a result of the overlaying, the com- 
posite image in which text subtitles are overlaid on video 
is obtained. ^5 
[01 1 8] The compositor 29 overlays the uncompressed 
graphics object rendered on the interactive graphics 
plane 26 with the composite image (uncompressed pic- 
ture data overlaid with the uncompressed graphics object 
rendered on the presentation graphic plane 18) output 20 
from the compositor 28. 

The switch 30 selectively supplies the TS packets read 
from the BD-ROM or the TS packets read from the local 
storage to the transport buffer 20. 

[0119] The network device 31 is used to implement 25 
communications functionality of the playback device. 

More specifically, the network device 31 establishes TCP 
connection, FTP connection, and so on with a web site 
at an URL. The contents downloaded from the web site 
via the connection established by the network device 31 so 
are stored to the local storage 200. 
The switch 32 selectively supplies the TS packets read 
from the BD-ROM or the TS packets read from the local 
storage to the transport buffer 12. 

[0120] The source de-packetizer 34 removes the TP_ 35 
extra_header from each TS packet constituting the AV 
clip read from the local storage 200 and outputs the TS 
packets without headers to the PID filter 35. The output 
of the TS packets to the PID filter 35 is carried out at the 
timing when the time measured by the arrival time clock 
counter 33 reaches the ATS shown by the TP_extra_ 
header. 

[0121] The PID filter 35 switches to output the TS pack- 
ets read from the local storage 200 toward either of the 
PG stream decoder, IG stream decoder, the audio de- ^5 
coder 39, and the audio decoder 42. 
The switch 36 supplies toward the audio decoder 39 the 
TS packets read from the BD-ROM or from the local stor- 
age 200. The TS packets constitute the primary audio 
stream. By the presence of the switch 36, the primary 50 
audio stream can be supplied to the audio decoder 39 
from either of the BD-ROM and the local storage 200. 
[01 22] The transport buffer (TB) 37 stores TS packets 
carrying the primary audio stream. 

The elementary buffer (EB) 38 stores the PES packets 55 
carrying the primary audio stream. 
The audio decoder 39 decodes the PES packets carrying 
the primary audio stream output from the elementary 



buffer 41 and outputs uncompressed audio data. 
[01 23] The transport buffer (TB) 40 stores TS packets 
carrying the secondary audio stream. 
The elementary buffer (EB) 41 stores PES packets car- 
rying the secondary audio stream. 
The audio decoder 42 decodes the PES packets carrying 
the secondary audio stream output from the elementary 
buffer 38 and outputs uncompressed audio data. 
[0124] The mixer 43 mixes the uncompressed audio 
data obtained by decoding the primary audio stream, with 
the uncompressed audio data obtained by decoding the 
secondary audio stream. The mixer 43 then outputs the 
composite audio resulting from the mixing. 
The scenario memory 44 stores current PlayList infor- 
mation and current Clip information. The current PlayList 
information used herein refers to the currently processed 
PlayList information from among a plurality of pieces of 
PlayList information stored on the BD-ROM. The current 
Clip information used herein refers to the currently proc- 
essed Clip information from among a plurality of pieces 
of Clip information stored on the BD-ROM. 
[01 25] The controller 45 is composed of an instruction 
ROM and CPU and executes software stored on the in- 
struction ROM to carry out overall control of the playback 
device. The control executed on the playback device dy- 
namically changes in response to a user event generated 
upon receipt of a user operation and in accordance with 
the values held in each PSR of the PSR set 49. The 
controller 45 is composed of functional elements of the 
main converter 46, the sub-converter 47, and the PL play- 
back controller 48. 

[0126] The main converter 46 converts playback 
points on the Playltem timeline to corresponding ad- 
dresses on the Main Clip. The playback points for starting 
normal playback and trick play are defined on the Play- 
ltem timeline. With reference to the EP_map included in 
the Clip information associated with the Main Clip, the 
main converter 4 6 converts the playback points to the 
corresponding addresses on the Main Clip. 
The sub-converter 47 first converts the playbackpoints 
on the Playltem timeline to corresponding playback 
points on the SubPlayltem timeline, and further converts 
the resulting playback points to corresponding addresses 
on the SubClip. The conversion of the playback points 
by the sub-converter 47 is carried out with reference to 
the sync_Playltem_id and the sync_start_PTS_of_P lay- 
Item included in the SubPlayltem information. 
[0127] The playback points on the SubPlayltem time- 
line are converted to corresponding addresses on the 
SubClip through the following two processes: 

(i) determining the entry point that is near the random 
access point form among a plurality of entry points 
shown by the EP_map; and 

(ii) performing the stream analysis starting from the 
thus determined entry point. 

Note that the process (i) is carried out using the EP_map 



16 



31 



EP 1 715 686 A1 



32 



for the SubClip. The process (ii) (i.e., stream analysis) is 
necessary for allowing random access in the case where 
the SubClip is the primary or secondary audio stream. In 
the case where the SubClip is the PG or IG stream, how- 
ever, the stream analysis may be omitted and random 
access to the SubClip is still ensured. Whether it is pos- 
sible to omit the process of stream analysis is determined 
with reference to the EP_stream_type. The stream anal- 
ysis may be omitted because the Clip information con- 
tains EP map that ensures the stream to be suitably ac- 
cessed at random. 

[01 28] The PL playback controller 48 carries out over- 
all control of the playback device for PL playback. The 
PL playback used herein refers to synchronous playback 
of the Main Clip stored on the BD-ROM and the SubClip 
stored on the local storage, in accordance with the Play- 
Item information and SubPlayltem information contained 
in the PlayList information. The Main Clip and SubClip 
are associated with their own EP_map, so that high- 
speed random access to the respective AV clips is en- 
sured. During PL playback, the "normal playback" and 
the "trick play" are implemented through high-speed ran- 
dom access to the respective clips. 
[0129] The normal playback during PL playback used 
herein refers to playback of specific portions of the Main 
Clip and SubClip on the Playltem timeline and the Sub- 
Playltem timeline, respectively. The portions of the re- 
spective clips are specified by the Playltem information 
defining the MainPath information and SubPlayltem in- 
formation defining the SubPath information. 
The trick play during PL used herein refers to fast-forward 
playback, rewind playback, chapter-search playback, 
and time-search playback of specific portions of the Main 
Clip and SubClip. The portions of the respective clips are 
specified by the Playltem information defining the Main- 
Path information and the Playltem information defining 
the SubPath information. 

[01 30] The PSR set 49 is a set of non-volatile registers 
provided within the playback device. The set of registers 

include 64 player status registers (PSR(1 )-PSR(64)) and 
4,096 general-purpose registers (GPRs). The 64 player 
status registers (PSRs) represent the current status of 
the playback device, including the current playback point. 
The current playback point is represented by the values 
of PSR(5)-PSR(8). Specifically, PSR(5) is set to a value 
from 1-999 to indicate the chapter number to which the 
current playback point belongs. When set to "OxFFFF", 
PSR(5) indicates that the chapter numbers are invalid in 
the playback device. 

[0131] PSR(6) is set to a value from 0-999 and indi- 
cates the PL number to which the current playback point 
belongs (current PL Number). 

PSR (7) is set to a value from 0-255 and indicates the 
Playltem number to which the current playback point be- 
longs (current PI Number). 

PSR(8) is set to a value from 0-OxFFFFFFFF and indi- 
cates the current playback point (current PTM) in 45 kHz 
accuracy. 



[0132] This concludes the description of the internal 
structure of the playback device. 

<Software Implementation: 1. PL Playback Controller 
48> 

5 Hereinafter, a description is given of software implemen- 
tation of the PL playback controller 48. The PL playback 
controller 48 can be implemented as computer software 
by creating a program for causing a computer to execute 
the processing steps illustrated in Fig. 30. Both the nor- 
10 mal playback and trick play during PL Playback are ex- 
ecuted through a jump playback in which playback is 
started at an arbitrary point on the Playltem timeline. 
Thus, the PL playback is executed by instructing the play- 
back device to execute jump playback. 
15 [0133] Fig. 30 is a flowchart illustrating the processing 
steps executed by the playback device for jump playback 
in which playback is started from an arbitrary point on 
the Playltem timeline. First, the playback device calcu- 
lates an offset a that indicates a coordinate of a point on 
20 the Playltem timeline (Step SI ). Next, the offset a is con- 
verted to a coordinate of a point on the MainClip timeline 
(ln_time + offset a) (Step S2). Then, the coordinate on 
the MainClip timeline (ln_time + offset a) is converted to 
a corresponding address a using the EP_map associat- 
es ed with the Main Clip (Step S3). 

[0134] In a step S4, the playbackdevice judges wheth- 
er the EP_stream_type in the EP_map associated with 
the SubClip is set to any of the values 3, 4, 6, and 7. If 
the EP_stream_type is not set to any of the above values, 
30 the playback device reads the BD-ROM starting from the 
address a (step S9). On the other hand, if the EP_ 
stream_type is set to any of 3, 4, 6, and 7, the playback 
device converts the offset a to an offset |3 (= offset a - 
sync_start_PTS_of_Playltem) indicating a coordinate of 
35 a corresponding point on the SubPlayltem timeline (step 
S5), and then converts the offset p to a coordinate of a 
corresponding point on the SubClip timeline 
(SubPlayltem_ln_time + offset |3) (Step S6). Then, with 
reference to the EP_map, the playbackdevice converts 
"^0 the thus obtained coordinate (SubPlayltem_ln_time + 
offset 3) to a corresponding address p (step S7). Then, 
the playback device simultaneously reads the BD-ROM 
starting from the address a and the local storage 200 
starting from the address p (step S8). 
45 [0135] If the EP_stream_type is set to a value other 
than 3, 4, 6, and 7 (step S4 : No), it means that the EP_ 
stream_type is invalid. When the EP_stream_type is 
invalid, the playbackdevice is provided with no informa- 
tion as to the intervals between entry points or the inter- 
so vals between entry times in the Out-of-MAX stream. With 
this being the situation, it is possible that it takes long to 
determine an access point on the SubClip and thus takes 
long to make random access to the SubClip. In such a 
case, synchronous playback of the Main Clip and SubClip 
55 may result in significant delay in the Main Clip playback. 
In order to avoid such undesirable possibility, the play- 
backdevice prohibits random access to the SubClip, and 
only reads the Main Clip starting from the address a (step 
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S9). Since random access is made only to the Main Clip, 
tliere will be no significant processing delay due to ran- 
dom access. 

[0136] Fig. 31 schematically illustrate how a random 
access point is determined using the EP map illustrated 5 
in Fig. 25. The following describes, with reference to Fig. 
31 , the processing steps performed to specify the random 
access point Illustrated in Fig. 31 . Note that Fig. 31 illus- 
trates the case where the requested jump point happens 
to coincide with an entry point of both the Main Clip and 
SubClip. Suppose that the requested jump point is at the 
offset a on the Playltem timeline, the origin point of the 
PlayList timeline is located at the point on the MainClip 
timeline specified by ln_time. Thus, the jump point on the 
Main Clip is calculated by "ln_time + offset a". Thus the ^5 
calculated result is converted to a corresponding SPN, 
and the SPN is further converted to a corresponding sec- 
tor number. 

[0137] On the other hand, the point on the SubPlay- 
Itemtimeline corresponding to the offset a on the Play- 20 
Item timeline is specified by sync_start_PTS_of_Play- 
Item + offset p. It is because the origin point of the Sub- 
Playltem timeline is apartfrom the origin point of the Play- 
ltem timeline by the distance indicated by sync_start_ 
PTS_of_Playltem. Once the offset p is calculated in the 25 
above manner, the coordinate of the jump point on the 
SubClip timeline is calculated. Since the origin point of 
the SubPlayltem timeline is located at the point on the 
SubPlayltem timeline specified by ln_time, the jump point 
is calculated by "SubPlayltem ln time + offset p". Then, so 
the jump point is converted into a corresponding SPN 
using the EP_map of the SubClip, and the SPN is con- 
verted to a corresponding address. 
<Software Implementation: 2. Main Converter 46 & Sub- 
Converter 47> 35 
Hereinafter, a description is given of software implemen- 
tation of the main converter 46 and sub-converter 47. 
The main converter 46 and sub-converter 47 can be im- 
plemented as computer software by creating a program 
for causing a computer to execute the processing steps 
illustrated in Fig. 32. 

[0138] Fig. 32 is a flowchart illustrating the processing 
steps executed by the playback device for converting a 
coordinate TM on the Main Clip and the SubClip to cor- 
responding addresses. ^5 
The playback device keeps adding the time length spec- 
ified by each PTS_EP_High included in the EP_High until 
the total E of the time lengths first exceeds the ln_time, 
to see how many PTS_High fields are added (Step S1 1 ). 
The time length indicated by each PTS_EP_High is a 50 
time unit whose most significant bits are indicated by the 
PTS_EP_High. The playback device then identifies the 
EP_High_id of the k-th EP_High that results in the total 
Z exceeding the ln_time and stores the value of k (step 
S12). 55 
[0139] Next, the playback device keeps adding, to the 
total Z of up to the PTS_EP_High (k-1), the time length 
indicated by each PTS_EP_Low included in EP_Low un- 
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til the total first exceeds ln_time, to see how many PTS_ 
Low fields can be added (step SI 3). The playback device 
then determines the EP_Low_id of the h-th EP_Low that 
results in the total exceeding the ln_time and stores the 
value of h (step S12). 

Using the pair of values "k-1" and "h-1" obtained as 
above, the playback device specifies the entry point that 
is near the ln_time. Using the SPN_EP_High and SPN_ 
EP_Low of the thus specified entry point, the playback 
device calculates the SPN that coincides with the coor- 
dinate TM and then converts the SPN to a corresponding 
sector address (step SI 5). 

[01 40] Conversion of SPN into a corresponding sector 

address is performed through the following processes. 
SPNs are serial numbers uniquely assigned to individual 
TS packets. Thus, for a playback device to read the TS 
packet corresponding to a specific SPN, it is necessary 
to convert the SPN to a relative sector number. Here, as 
illustrated in Fig. 4, every 32 TS packets are converted 
into one Aligned Unit and recorded into three sectors. 
Thus, the result obtained by dividing the SPN by 32 is 
the number assigned to the Aligned Unit containing an 
l-picture. Then, the result obtained by multiplying the thus 
obtained Aligned Unit number by 3 is the sector address 
of the Aligned Unit that is near the SPN. The sector ad- 
dress calculated in the above manner is a relative sector 
number serially assigned to each sector from the head 
of one AV clip. Thus, by setting the file pointer to the 
relative sector number, the sector corresponding to the 
entry point is specified. 

[0141] In a step SI 6, it is determined whether the EP_ 
stream_type of the EP_map for the SubClip is set to any 
of the values 3, 4, 6, and 7. In the case where the EP_ 
stream_type is set to 6 (PG stream) or 7 (IG stream), it 
means that the entry point near the coordinate TM coin- 
cides with the head (PCS or ICS) of a DS other than 
Normal Case DS. Since such a DS contains all the func- 
tional segments necessary for presenting a composite 
image, the address obtained by the conversion in the 
step SI 5 is designated as the address p, which is the 
access point (step S17). 

[01 42] Once the address p is determined, the playback 
device starts playback from the address p to duly present 
text subtitles or menus. 

In the case where the EP_stream_type is set to 3 (primary 
audio stream) or 4 (secondary audio stream), the play- 
back device performs stream analysis starting from the 
address obtained by the conversion in the step SI 5 (step 
S18). In the stream analysis, the PES packet header is 
extracted and the audio frame size and PTS are read 
from the header. The point at which the next audio frame 
is recorded is identified based on the size. The above 
process is repeated until the thus determined point reach- 
es the PES packet with the PTS set to a value greater 
than the coordinate TM (step SI 9). 
[01 43] Once the PES packet with the PTS setto a value 
greater than the coordinate TM is identified, the address 
of the PES packet is determined as the address p, which 
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is the access point (step S20). Tine playback device then 
starts playback from the address p to duly present audio. 
Hereinafter, the processing steps illustrated in the flow- 
chart are more specifically described with reference to 
Figs. 33 and 34. 

[0144] Fig. 33 illustrates the relationship among the 
variables k and h and the random access point, in the 
case where the SubClip is a primary or secondary audio 
stream. In the figure, the first level illustrates the PTS_ 
EP_High fields included in the EP_map. The second rev- 
el illustrates the PTS_EP_Low fields also included in the 
EP_map. The third level illustrates TS packets. In the 
figure, the k-th PTS EP High has the minimum value 
exceeding the random access point, and the h-th PTS_ 
EP_Low has the minimum value exceeding the random 
access point. 

[0145] Thus, the pair of PTS_EP_High[k-1] and the 
PTS_EP_Low[h-1 ] specifies the entry time near the ran- 
dom access point. 

Since the above pair of the PTS_EP_High and the PTS_ 
EP_Low specifies the entry point near the random access 
point, the PES packet corresponding to the random ac- 
cess point is specified through the stream analysis start- 
ed from the thus specified entry point. 
[0146] Fig. 34 illustrates the relationship among the 
variables k and h and the random access point, in the 
case where the SubClip is a PG or IG stream. In the 
figure, the first level illustrates the PTS_EP_High fields 
included in the EP_map. The second revel illustrates the 
PTS_EP_Low fields also included in the EP_map. The 
third level illustrates TS packets. In the figure, the k-th 
PTS_EP_High has the minimum value exceeding the 
random access point, and the h-th PTS_EP_Low has the 
minimum value exceeding the random access point. 
[0147] Thus, the pair of PTS_EP_High[k-1] and the 
PTS_EP_Low[h-1 ] specifies the entry time near the ran- 
dom access point. 

As stated above, the above pair of the PTS_EP_High 
and the PTS EP Low specifies the entry point near the 
random access point. Thus, as long as the playback de- 
vice starts reading from the thus specified entry point, it 
is always ensured that a DS composed of a complete set 
of functional segments is read. 

[0148] As described above, according to the present 
embodiment, the EP_map for the SubClip includes the 
EP_stream_type. The provision of EP_stream_type al- 
lows the playback device to be informed of whether the 
EP_map for the Out-of-MUX stream shows entry points 
set at constant time intervals or addresses of independ- 
ent playback sections. Being informed of the type of the 
EP_map, the playback device is enabled to instantly 
judge whether high-speed random access to the SubClip 
is possible. With this arrangement, synchronous play- 
back of the Main Clip and SubClip is executed without 
sacrificing a response performance. Thus, even during 
PlayList playback defining the Main Path + SubPath, 
jump playback is executed at the same level of response 
speed as that of jump playback executed solely on the 



Main Path. 

(SECOND EMBODIMENT) 

5 [0149] A second embodiment of the present invention 
relates to improvements in Picture in Picture (PiP). The 
PiP playback refers to playback of such PlayList infor- 
mation that includes Main Path information and SubPlay- 
Item information. In addition, the MainPath information 

10 specifies a Main Clip that contains video, and the Sub- 
Path information specifies a SubClip that contains differ- 
ent video. When PiP playback of the PlayList information 
is executed, video playback of the SubClip (secondary 
video) is presented within video playback of the Main Clip 

15 (primary video) on the same screen. 

[01 50] Fig. 35 illustrates one example of PiP playback. 
In the figure, until the current playback point reaches a 
predetermined point tp, playback of the primary video is 
solely presented. After the point tp is reached, playback 

20 images of the secondary video are presented within play- 
back images of the primary video. In this example, the 
primary video is presented with HD (high-definition) im- 
ages, whereas the secondary video is presented with SD 
(standard-definition) images. 

25 In this example, the secondary video presents video im- 
ages in which the director or a performer of a movie points 
with his finger a specific object appearing in the primary 
video. When the secondary video is presented in com- 
bination with the primary video playback, the combined 

30 images present a scene in which the director or the per- 
former points out with his finger the specific object ap- 
pearing in playback images of the primary video and 
gives commentary about the object. 
[0151] Fig. 36A illustrates an HD image and an SD 

35 image for comparison. 

The HD image has the resolution of 1920 x 1080. The 
frame interval is a 3750 clock (alternatively, a 3753 or 
3754 clock), which is equivalent to the frame interval of 
a theatrical movie film. 

40 The SD image is has the resolution of 720 x 480. The 
frame interval is a 1501 clock, which is equivalent to the 
frame interval of NTSC. Alternatively, the frame interval 
may be an 1800 clock, which is equivalent to the frame 
interval of PAL. 

45 [0152] As illustrated in the figure, the SD image reso- 
lution is about 1/4 of the HD image resolution. Thus, when 
the primary video (i.e. HD images) and the secondary 
video (i.e. the SD images) are presented on the same 
screen, thesizeof the secondary video on screen is about 

50 1 /4 of the primary video. Fig. 36B illustrates how the sec- 
ondary video is scaled up and down. The secondary vid- 
eo is scaled up and down in accordance with the scaling 
factor. The scaling factor defines xl/4 height, xl/2 
height, x 1 .5 height, or x2.0 height. The playback device 

55 reduces or enlarges the secondary video in height ac- 
cording to the scaling factor. At the same time, the play- 
back device also reduces or enlarges the secondary vid- 
eo widthwise, so as to maintain the aspect ratio of the 
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original SD image. 

[01 53] Through the scaling up and scaling down of the 
secondary video, the layout of the secondary video in 
PiP playback is freely changed. 

Now, the following describes the structures of the record- 5 
ing medium and the playback device for creating a PiP 
playback application described above. In this embodi- 
ment, the AV clip stored on the BD-ROM constitutes pri- 
mary video. It is simply because a large volume medium 
is suitable for distribution of the primary video having a '^o 
large volume of data. On the other hand, the SubClip 
constituting secondary video is distributed to the play- 
back device via a network and stored in the local storage, 
along with the PlayList information defining synchronous 
playback of the primary video and the secondary video. ^5 
[0154] <Structure of Local Storage 200: 1 . File Struc- 
ture> 

Fig. 37 illustrates the contents stored on the local storage 
according to the second embodiment. The structure of 
the local storage is illustrated in Fig. 37 in the similar 20 
manner to Fig. 1 1 . The difference with Fig. 1 1 is found 
in that the SubClip (00002. m2ts) is a secondary video 
stream and the clip information (00002. clip) contains the 
EP_map for the secondary video stream. 
<Structure of Local Storage 200: 2. Clip lnformation> 25 
Fig. 38 illustrates the internal structure of the clip infor- 
mation stored on the local storage. The clip information 
is associated with the secondary video stream. In the 
figure, leader lines cu2, cu3, cu4, and cu5 indicate that 
the internal structure of the EP_map included in the clip 30 
information is illustrated in more detail. 
[0155] The EP_map pointed to by the leader lines is 
identical in structure to the EP_map illustrated in Fig. 8. 
The EP_map of the secondary video shows entry points 
set on the head of each access unit (GOP) constituting 35 
the secondary video stream, and also shows entry times 
corresponding to the entry points. Although for the sec- 
ondary video, the EP _map is still set for video. Thus, the 
time interval between adjacent entry times is less than 
one second, and the EP_stream_type is set to the value 
"1 : Video Type" similarly to the example shown in Fig. 8. 
[0156] The leader lines ct1 and ct2 indicate that the 
internal structure of Clip Info is illustrated in more detail. 
In the Clip Info, the application_type is set to the value 
indicating that the SubClip associated with the clip infer- ^5 
mation is "TS for additional content with video". Fig. 39 
illustrates the EP_map set for the secondary video 
stream in the same manner as Fig. 9. 
[01 57] This concludes the description of the clip infor- 
mation according to the second embodiment. 50 
<Structure of Local Storage 200: 3. PlayList lnformation> 
Next, the following describes the PlayList information ac- 
cording to the second embodiment. There are two types 
of PiP playback; one is statically synchronized playback 
and the other is dynamically synchronized playback. The 55 
above-mentioned example is a statically synchronized 
PiP application for synchronously presenting the main 
feature movie with the commentary, which are the pri- 



mary video and the secondary video, respectively. 
[0158] <Details of PlayList information: 1. Compo- 
nents for Implementing Static PiP Playback> 
Fig. 40 illustrates PlayList information defining statically 
synchronized PiP playback. The PlayList information de- 
fining the statically synchronized PiP playback may in- 
clude a plurality of pieces of SubPath information (Mult- 
SubPath) and each piece of SubPath information may 
include a plurality of pieces of SubPlayltem information 
(IVlulti-SubPlayltem). 

[0159] The SubPlayltem information illustrated in the 
figure additionally includes a PiP_position field and a 
PiP size field. Each field included in the SubPlayltem 

information is set as follows. 

The "Clip_information_file_name" field is set to indicate 
the file name of the AV clip containing the primary video. 
The "sub Playltem ln time" field is set to the value in- 
dicating the start point of the SubPlayltem on the time 
line of the SubClip containing the secondary video. 
[01 60] The "sub_Play ltem_Out_time" field is set to the 
value indicating the end time of the SubPlayltem on the 
timeline of the SubClip containing the secondary video. 
The "sync_Playltem_id" field is associated with the AV 
clip containing the primary video and set to the value 
indicating the Playltem identifier. 
The "sync_start_PTS_of_P lay Item" field is timing infor- 
mation indicating the timing at which playback of the play- 
back section (SubPlayltem) defined by the SubPlayltem 
information is to be started. Here, the playback start time 
of SubPlayltem is expressed using the number of sec- 
onds to be lapsed from the playback start of the Playltem 
defined by the sync_Playltem_id. 
[0161] The "PiP_position" indicates the X and Y coor- 
dinates of a position in the screen plane at which the 
secondary video playback is to be presented. 
The "PiP_size" indicates the height and the width of the 
secondary video playback. 

<Details of PlayList Information: 2. Synchronization in 
Statically Synchronized PiP Playback> 
Fig. 41 illustrates how the PlayList information defines 
the synchronization between the Main Clip containing 
the primary video and the SubClip containing the sec- 
ondary video. Fig. 41 is illustrated in the same manner 
as Figs. 25 and 26. 

[0162] The SubClip containing the secondary video is 
provided with an EP_map and thus the high-speed ran- 
dom access to the SubClip is ensured in time accuracy 
of less than one second. The PlayList information defines 
synchronous playback of the Main Clip and SubClip on 
precondition that high-speed random access to the Sub- 
Clip is possible. 

Here, the following describes the technical significance 
of the provision of EP_map for a SubClip containing sec- 
ondary video. The SubClip containing secondary video 
is provided with an EP_map because trick play of the 
secondary video needs to be executed in response to 
trick play of the primary video. In the case where trick 
play of the primary video is executed during PiP playback. 
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how to process the secondary video matters. During fast- 
forwarding or rewinding of the primary video, it is possible 
to continually play backthe secondary video in the normal 
mode. Yet, In such a case, the playback device needs to 
have separate decoders for the primary video and the 
secondary video, and the decoders need to be operated 
on separate system time clocks (STCs). It is impractical 
to provide two STCs to a player model primarily Intended 
for home use. 

[01 63] Practically, the secondary video is fast-forward- 
ed and rewound in synchronism with fast-forwarding and 
rewinding of the primary video. In view of this, the EP_ 
map is also provided for the SubCllp so as to allow the 
secondary video to be fast-forwarded and rewound in 
synchronism with fast-forwarding and rewinding of the 
primary video. 

This concludes the description of the PlayLlst information 
defining statically synchronized playback of the Main Clip 
and SubClip. 

<Details of PlayLlst Information: 3. Synchronization In 
Dynamically Synchronized PIP Playback> 
Next, a description is given of the PlayLlst information 
defining dynamically synchronized playback of the Main 
Clip and SubClip. 

[0164] The dynamic synchronization refers to play- 
back of the Sub Play Item associated with the secondary 

video stream (SubClip) synchronously with playback of 
the primary video stream (Main Clip), and the playback 
start point of the SubPlayltem Is dynamically determined 
in response to a user operation. For example, the author 
of a PiP application intends to implement such dynamic 
synchronization as illustrated in Figs. 42A through 42C. 
[0165] Fig. 42A illustrates the display screen present- 
ed during playback of the primary video of the PIP appli- 
cation. Each button on the screen illustrated in Fig. 
42Aispresented using a thumbnail image (thumbnail but- 
ton). Each thumbnail button has three states that are a 
normal state, a selected state, and an activated stated. 
The PIP application Is designed so as to present, when 
one of the thumbnail buttons receives the selected state 
(Fig. 42B), playback image of the secondary video in 
place of the thumbnail image used to present the selected 
button (Fig. 42C). As In this example, the playback start 
point of the secondary video is dynamically determined 
in response to a user operation made during playback of 
the primary video. This is what Is referred to as "dynamic 
synchronization". A user operation for changing a thumb- 
nail button to the selected state (in this case, a user op- 
eration for selecting the thumbnail button) is referred to 
as a rock operation. 

[0166] In the case of dynamic synchronization, It Is not 
known in advance which of the thumbnail images will be 
selected and thus which secondary video is to be played 
back. Thus, it is necessary to dynamically determine, dur- 
ing playback of the primary video, the start point of the 
synchronous playback of the secondary video. 
<Details of PlayLlst Information: 4. Components for Im- 
plementing Dynamically Synchronized PiP Playback> 



Fig. 43 illustrates the internal structure of the PlayLlst 
information defining dynamically synchronized PIP play- 
back. 

[0167] The PlayLlst Information defining the dynaml- 

5 cally synchronized PiP playback may include a plurality 
of pieces of SubPath information (Mult-SubPath). Yet, 
each piece of SubPath information can include a single 
piece of SubPlayltem Information (SIngle-SubPlayltem). 
Similarly to the example Illustrated In Fig. 40, the Sub- 

10 Playltem information illustrated in the figure includes a 
PiP_position field and a PiP_size field. The setting of 
"Cllp_lnformatlon_flle_name", "SubPlayltem_ln_tlme", 
"SubPlayltem_Out_tlme", "sync_Playltem_id", "PIP_po- 
sition", and "PiP_size" fields is identical to the setting of 

15 corresponding fields illustrated in Fig. 40. The difference 
lies in the setting of the "sync_start_PTS_of_P lay Item". 
[0168] The "sync_start_PTS_of_Playltem" Indicating 
a point for establishing synchronous playback with the 
Playltem specified by the sync_Playltem_id is set to an 

20 undefined value. If set to an undefined value, the "sync_ 
start_PTS_of_Playltem" Indicates that the synchroniza- 
tion point is dynamically determined, in response to a 
rock operation by a user, on the timeline of the Playltem 
specified by the sync_Playltem_ld. 

25 In accordance with the PlayLlst Information Illustrated In 
Fig. 43, the playback device operates to write the point 
of a rock operation to the sync_start_PTS_of_Playltem. 
Here, if the point of rock operation is defined as a time 
at which a thumbnail button Is first changed to the select- 
so ed state and then automatically to the activated state, 
the playback device starts playback of the secondary vid- 
eo at the time when the thumbnail button is changed to 
the activated state. Since this press relates specifically 
to the selection and automatic activation of thumbnail 

35 buttons, it is desirable to define this process using a navi 
command, which is used to control the behavior of thumb- 
nail buttons. 

[0169] This concludes the description of the Internal 
structure of the PlayLlst Information Implementing dy- 

^0 namically synchronized PiP playback. This concludes 
the description of the improvements made on the record- 
Ingmedlum, according to the present embodiment. 
Now, the following describes Improvements made on the 
playback device, according to this embodiment. 

^5 <Structure of Playback Device: 1 . Overall Structure> 
Fig. 44 Illustrates the Internal structure of the playback 
device according to the second embodiment. For the 
sake of convenience in illustration, the components of 
the audio decoder are omitted. 

50 [0170] Fig. 44 is based on the structure of the playback 
device Illustrated In Fig. 29, and the same reference nu- 
merals are used to denote the same components. Among 
the components in common with the playback device il- 
lustrated In Fig. 29, the transport buffer 5, the multiplexed 

55 buffer 6, the coded picture buffer 7, the video decoder 8, 
the decoded picture buffer 10, and the primary video 
plane 1 1 in the second embodiment function to decode 
the primary video stream. 
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[01 71 ] On the other hand, for decoding the secondary 
video stream, the playback device illustrated in Fig. 44 
additionally includes a transport buffer 51 , a multiplexed 
buffer 52, a coded picture buffer 53, a video decoder 54, 
a decoded picture buffer 55, a secondary video plane 56, 
a scaler 57, and a compositor 58. Hereinafter, a descrip- 
tion of those additional components will be made. 
<Structure of Playback Device 300: 2. Additional Hard- 
ware Components> 

The transport buffer 51 is a buffer for temporarily storing 
TS packets carrying the secondary video stream (Sub- 
Clip) output from the PID filter 35. 
[0172] The multiplexed buffer (MB) 52 is a buffer for 
temporarily storing PES packets output from the trans- 
port buffer 51 , in order to later output the secondary video 
stream to the coded picture buffer 53. 
The coded picture buffer (CPB) 53 is a buffer for storing 
encoded pictures (I, B, and P pictures). 
[0173] The video decoder 54 decodes individual 
frames contained in the secondary video stream at every 
decoding time shown by a decoding time stamp (DTS) 
to obtain a plurality of frames and renders the resulting 
picture data on the decoded picture buffer 55. 
The decoded picture buffer 55 is a buffer on which de- 
coded picture data is rendered. 

The secondary video plane 56 is used for presenting un- 
impressed picture data obtained by decoding the sec- 
ondary video. 

[0174] The scaler 57 is for reducing or enlarging the 
size of playback image rendered on the secondary video 

plane 56, in accordance with the height and the width 
indicated by the PiP_size field included in the SubPlay- 
Item information. 

The compositor 58 is for implementing PiP playback by 
overlaying the playback image resized by the scaler 57, 
on the playback image obtained by the video decoder. 
The overlaying of the playback images of the primary 
video and the secondary video is carried out in accord- 
ance with the PiP position defined in the SubPlayltem 
information. As a result, composite images of the primary 
and secondary video playback are presented. The com- 
positor 58 is capable of chroma key composition, layer 
composition, and so on. Thus, it is possible to remove 
the background image from the secondary video to only 
leave a film character and then overlay the resulting im- 
age of the film character with the playback image of the 
primary video. 

[01 75] With the above additional components, the PID 
filter 35 according to the second embodiment supplies 
TS packets carrying the secondary video stream to the 
transport buffer 51 , the multiplexed buffer 52, the coded 
picture buffer 53, the video decoder 54, the decoded pic- 
ture buffer 55, and the secondary video plane 56. 
<Structure of Playback Device 300: 2. Additional Hard- 
ware Component to Controller 45> 
Thecontroller45additionallyincludesasync-setting unit 
50. 

[0176] The sync-setting unit 50 judges whether the 



sync_start_PTS_of_Playltem included in the SubPlay- 
ltem information is set to an undefined value. If the sync_ 
start_PTS_of_Playltem is set to an undefined value, the 
playback device is brought into readiness to receive a 

5 rockoperation made by a user during the time a playback 
section of the Main Clip specified by the Playltem infor- 
mation is solely played back. The rock operation is re- 
ceived via a remote controller and determines the start 
point of synchronous playback. Upon receipt of the rock 

^0 operation, the undefined value of the sync_start_PTS_ 
of_Playltem is overwritten with the value identifying the 
time at which the rock operation is received. In the case 
where the rock operation is defined as a selection of any 
of buttons presented on the playback image of the Main 

^5 Clip, the time at which the operation of selecting the but- 
ton is made is regarded as the time of the rockoperation. 
[01 77] Once the value of the sync_start_PTS_of_P I ay- 
Item is set in the above manner, the PL playback con- 
troller 48 can control playback of the Playltem specifying 

20 the primary video and the SubPlayltem specifying the 
secondary video. As a result, dynamically synchronized 
PiP playback is executed. 

It should be naturally appreciated that if the value of the 
sync_start_PTS_of_P lay Item is statically set to a specific 
25 value, the PL playback controller 48 can control playback 
of the Playltem and the SubPlayltem so as to execute 
statically synchronized PiP playback. 
[01 78] <Structure of Playback Device: 2. Software Im- 
plementation> 

30 This concludes the description of the components addi- 
tionally provided in the second embodiment. Next, a de- 
scription is given of improvements made in the software 
implementation of the second embodiment. 
Fig. 45 is a flowchart illustrating processing steps for ex- 

35 ecuting jump playback starting from an arbitrary point on 
the Playltem timeline. The flowchart in Fig. 45 is based 
on the flowchart illustrated in Fig. 30 and is identical there- 
to except that the steps S4-S8 are replaced with steps 
S31 -S35. In the step S31 , it is judged whether the EP_ 

"^0 stream_type in the EP_map associated with the SubClip 
indicates the video type. If the video type: 1 is indicated 
(step S31 : YES), an offset a indicating a point on the 
Playltem timeline is converted into an offset (3 indicating 
a corresponding point on the SubPlayltem timeline (= 

45 offset a - sync_start_PTS_of_Playltem) (step S32). 
Then, the offset p is converted into a coordinate of a 
corresponding point on the SubClip timeline 
(SubPlayltem_ln_time + offset P) (step S33). Then, with 
reference to the EP_map, the SubPlayltem_ln_time + 

50 offset p is further converted into an address p of a corre- 
sponding GOP in the SubClip, which contains the sec- 
ondaryvideo (step S34). Then, the playback device si- 
multaneously accesses the BD-ROM at the address a 
and the local storage 200 at the address p to read GOPs 

55 of the primary video and GOPs of the secondary video, 
respectively (step S35). 

[0179] In the step S31 , if it is judged that the EP_ 
stream_type does not indicate the video type: 1 , the steps 
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S4-S8 illustrated in Fig. 30 are performed. 
Fig. 46 schematically illustrates, in the same manner as 
Fig. 31 , random access to the Main Clip and to the Sub- 
Clip. The SubClip containing the secondary video has 
entry points at time intervals of less than 1 sec. Random 
access to the SubClip is made using those entry points. 
Since random access to the SubClip can be made at 
about the same speed as random access to the IVIain 
Clip, jump playback of the SubClip is executed in syn- 
chronism with jump playback of the Main Clip. 
[0180] As described above, during PiP playback in 
which is HD images of the primary video are presented 
in combination with SD images of the secondary video, 
a random access is made at a relatively high speed. By 
virtue of this high-speed random access capability, trick 
play can be executed even during PiP playback. 
(Supplemental Notes) 

Needless to say, the foregoing descriptions do not cover 
all the modes of practicing the present invention. The 
present invention can be practiced also by any of the 
following modifications (A), (B), (C), (D), ... etc. It should 
be noted that the inventions recited in the claims of the 
present application are broadened or generalized de- 
scriptions of the above-described embodiments and their 
modifications. The extent of the broadening and gener- 
alization reflects the state of the art at the time of filing 
the present application. 
[0181] 

(A) Information processing performed by the func- 
tional components or shown in the flowcharts de- 
scribed in the above embodiments is actually real- 
ized using hardware resources. Accordingly, the in- 
formation processing is said to be a highly advanced 
creation of technical ideas by which a low of nature 
is utilized, and thus satisfies the requirements of 
"program invention". 

■ Program Production According to the Present In- 
vention 

A program according to the present invention may 
be produced in the following manner. First, a soft- 
ware developer writes, in a programming language, 
a source program for implementing the flowcharts or 
the functional components described above. When 
writing the source program for implementing the 
flowcharts or the functional components, the soft- 
ware developer may use class structures, variables, 
array variables, and calls for external functions, in 
accordance with the syntax of that programming lan- 
guage. 

[0182] More specifically, for example, each loop 
shown in the flowchart may be described using a FOR 
statement and each Judgment may be described using 
an IF statement or a SWITH statement, in accordance 
with the syntax. The hardware control including the play- 
back control of the decoder and the access control of the 



drive device can be described using CALL statements 
for external functions provided by the hardware manu- 
facturer. 

[0183] The resulting source program is supplied as a 

5 file to a compiler. The compiler translates the source pro- 
gram into an object program. 

The compilation involves processes of parsing, optimi- 
zation, resource allocation, and code generation. The 
parsing involves lexical analysis, syntactic analysis, and 
10 semantic analysis of the source program, and conversion 
of the source program into an intermediate program. The 
optimization involves processes of dividing the interme- 
diate program into basic blocks, and control flow analysis 
and data flow analysis of the intermediate program. The 
15 resource allocation involves a process of allocating reg- 
isters or memory of a target processor to variables ap- 
pearing in the intermediate program, whereby the inter- 
mediate program is adapted to the instructions sets of 
the processor. The code generation involves a process 
of converting intermediate statements of the intermediate 
program into program code, whereby an object program 
is generated. 

[0184] The thus generated object program is com- 
posed of one or more lines of code for causing a computer 
to perform processing steps shown in the flowcharts ref- 
erenced in the above embodiments or the processing 
steps performed by the functional components. The pro- 
gram code mentioned herein may be of any of a various 
types of codes including a native code of the processor 
and Java (registered trademark) bytecode. The process- 
ing steps may be implemented by the program code in 
various ways. For example, when a step is implemented 
using an external function, a CALL statement for the ex- 
ternal function comprises a line of the program code. In 
addition, there may be a case where program code for 
implementing one step is contained in two or more sep- 
arate object programs. When a RISC processor, which 
is designed for handling reduced instruction sets, is em- 
ployed, each processing step of the flowcharts may be 
implemented using a combination of an arithmetic in- 
struction, a logical instruction, and a branch instruction, 
for example. 

[01 85] Once the object program is generated, the pro- 
grammer activates a linker. The linker allocates memory 
areas for the object program and related library pro- 
grams, and binds them together to generate a load mod- 
ule. The thus generated load module is to be read by a 
computer thereby to cause the computer to perform the 
processing steps shown in the above flowcharts or the 
processing steps performed by the functional compo- 
nents. Through the above processes, a program embod- 
ying the present invention is created. 

(B) The program according to the present invention 
may be used in the following manners. 

[0186] 
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(i) As Embedded Program 

When the program according to the present invention is 
used as an embedded program, a load module that is 
equivalent to the program is recorded into an instruction 5 
ROM, along with a basic input output program (BIOS) 
and various pieces of middleware (operation systems). 
The instruction ROM is then built into the control unit and 
executed by the CPU. As a result, the program according 
to the present invention is used as the control program 10 
of the playback device. 
[0187] 

(ii) As Application Program 

15 

In the case where a playback device is provided with an 
internal hard disk, a Basic Input/Output System (BIOS) 
is already embedded within the instruction ROM and mid- 
dleware (operation system) is preinstalled onto the hard 
disk. In addition, the playback device is provided with a 20 
boot ROM for activating the system from the hard disk. 
[0188] Thus, in this case, the load module is supplied 
to the playback device via a portable recording medium 
or a network, and the load module is installed as a single 
application program onto the hard disk. Then, the play- 25 
back device bootstraps to activate the operation system 
and instructs the CPU to execute the application. In this 
way, the program according to the present invention is 
used as a single application program. 
[0189] As described above, with a playback device so 
having an internal hard disk, the program according to 
the present invention can be used as a single application 
program. Thus, the program according to the present in- 
vention may be singly transferred, leased, or supplied 
over a network. 35 

(C) Production and Usage of System LSI according 
to the Present Invention 

Generally, a system LSI is composed of a bare chip pack- 
aged on a high-density substrate. Alternatively, a system 
LSI may be composed of a plurality of bare chips that is 
packaged on a high-density substrate and has an exter- 
nal structure just as a single LSI (this type system LSI is 
referred to as a multichip module). ^5 
[0190] Focusing on the types of packaging, there are 
different types of system LSIs called QFP (quad flat pack- 
age) and PGA (Pin Grid Array). QFP is a type of system 
LSI with pins extending from all four sides of the package. 
PGA is a type of system LSI package with an array of 50 
pins that are arranged on entire surface of the base of 
the package. 

The pins act as an I/O interface with a drive device, a 
remote controller, and a television. The pins also act as 
an IEEE1394 interface and a PCI bus interface. Since 55 
the pins of the system LSI act as interface, by connecting 
circuitry of a drive device or a playbackdevice, the system 
LSI plays a roll as the core of the playback device. 



46 

[0191] A bare chip packaged into a system LSI may 
be an instruction ROM, CPU, a decoder LSI implement- 
ing the functions of the components shown in the figures 
of the internal structures according to the above embod- 
iments. 

As mentioned in the description given in relation to the 
use "As Embedded Program", the instruction ROM stores 
a load module equivalent to the program of the present 
invention, a BIOS, and various pieces of middleware (op- 
eration systems). The part to which the above embodi- 
ments are particularly relevant is the load module equiv- 
alent to the program. Thus, by packaging, as a bare chip, 
the instruction ROM storing the loadmodule equivalent 
to the program, a system LSI according to the present 
invention is produced. 

[01 92] The details of the procedure for producing such 
system LSIs are as follows. The first step is to make a 
circuit diagram of a part to be incorporated into a system 
LSI, based on the figures showing the internal structures 
according to the above embodiments. The next step is 
to embody each component shown in the figures using 
circuit components, ICs, and LSIs. 
The next step following the embodying each component 
is to design a bus connecting circuit elements, ICs, and 
LSIs, the peripheral circuitry, and interface with external 
devices. In addition, connecting lines, power lines, 
ground lines, and clock signal lines are designed. In this 
process, operation timing of each component is adjusted 
in consideration of the LSI spec. In addition, some adj 
ustment is made to ensure the bandwidth of each com- 
ponent. In this way, the circuit diagram is completed. 
[0193] Once the circuit diagram is ready, the packag- 
ing design needs to be made next. The packaging design 
is a process of designing a layout on a substrate. To this 
end, the physical arrangement of the elements (circuit 
elements, IC, and LSI) shown in the circuit diagram is 
determined and the wiring on the substrate is also deter- 
mined. 

The packaging design mentioned herein may be done 
through the processes called auto layout and auto wiring. 
When a CAD apparatus is employed, the auto layout is 
carried out using a dedicated algorithm called "centroid 
method". In the process of auto wiring, the connecting 
lines between the elements and pins shown on the circuit 
diagram are implemented using metal foil and vias. With 
a CAD apparatus, the wiring is completed using any of 
dedicated algorithms called "maze routing algorithm" and 
"line-search algorithm". 

[0194] After the packaging design is completed and 
the layout on the substrate is determined, the related 
data is converted into CAM data and supplied to appro- 
priate devices such as an NC machine tool. The NC ma- 
chine tool incorporates the elements using System on 
Chip (SoC) or System in Package (SiP) implementations. 
According to the SoC implementation, multiple circuits 
are baked on a single chip. According to the SiP imple- 
mentation, multiple chips are joined into asingle package 
with, for example, resin. Through the above processes. 
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a system LSI according to the present invention can be 
produced based on the figures showing the internal struc- 
ture of the playback device referenced in the above em- 
bodiments. 

[0195] Note that integrated circuits produced in the 5 
above manner may be referred to as IC, LSI, super LSI, 
or ultra LSI, depending on the packaging density. 
In addition, all or some of the components of each play- 
back device of the present Invention maybe incorporated 
into a single chip. In addition, instead of the SoC and SiP 'io 
implementations, the elements may be integrated using 
a dedicated circuit or a general-purpose processor. For 
example, it is applicable to use an FPGA (Field Program- 
mable Gate Array) that can be programmed after an LSI 
is produced. It is also applicable to use a reconfigurable ^5 
processor that allows reconstruction of connection be- 
tween circuit cells within the LSI and their settings. When 
any newcircuit integration technology becomes available 
or derived as the semiconductor technology advances, 
such new technology may be employed to integrate the 20 
functional blocks of the present invention. One possible 
candidate of such new technology may be achieved by 
adapting biotechnology. 
[0196] 

25 

(D) According to the above embodiments, the re- 
cording medium according to the present invention 
is described as a hard disk. Yet, it should be noted 
that the features of the recording medium according 
to the present invention lie in the EP map and the so 
EP_stream map recorded thereon and these fea- 
tures do not depend on the physical property of the 
hard disk. Any other recording medium is applicable 
as long as the recording medium can store the EP_ 
map and the EP_stream_type and can be used in 35 
combination with a BD-ROM. Specific examples of 
such recording media include: semiconductor mem- 
ory cards, such as a CompactFlash card (registered 
trademark), a SmartMedia card, a Memory Stick 
card, a MultiMediaCard, and a PCM-CI A card. Alter- 
natively, the recording medium according to the 
present invention maybe (i) amagneticdisk, such as 
a flexible disk, SuperDisk, Zip, or Clik! or (ii) a re- 
movable hard disk drive, such as ORB, Jaz, SparQ, 
SyJet, EZFley, or Microdrive. 45 

[0197] 



an AV clip may be compliant with Dolby-AC3, MPS, 
MPEG-AAC, or dts. 

[0198] 

(F) According to the above embodiments, the video 
stream is an MPEG4-AVC stream (which may also 
be referred to as "H.264" or "JVT"). Yet, the video 
stream may be of MPEG2 format or any other format 
such as the VC-1 format as long as the video stream 
is solely decodable. 

(G) In the case where the value of sync_start_PTS_ 
of Playltem is set by the sync-setting unit 50, it is 
desirable that the sync-setting unit 50 perform the 
playback control as illustrated in Figs 47A-47C. 

[0199] Fig. 4 7A illustrates the playback control for ex- 
ecuting dynamically synchronized PiP playback. An ar- 
bitrary point on the Playltem timeline is determined as 
the start point for synchronization with the SubPlayltem. 
When the current playback point reaches the time indi- 
cated by the sync_start_PTS_of_P lay Item, the playback 
device starts decoding the secondary video and overlays 
playback of the secondary video with playback of the 
primary video. 

[0200] Fig. 478 illustrates dynamically synchronized 
PiP playback executed in the case where the rock point 
is once passed and reached again in the normal playback 
mode as a result of rewinding. In other words, the figure 
illustrates how dynamic synchronization in PiP playback 
is established in the case where the current playback 
point once passes the rock point in the normal playback 
mode and then is rewound by the user beyond the rock 
point, and reaches the rock point for the second time in 
the normal playback mode. 

[0201 ] When the rock point is reached for the first time, 
PiP playback is started. PiP playback is stopped at the 
moment when the rewinding operation is made. Then, 
playback is resumed in the normal playback mode and 
PiP playback is not executed even when the rock point 
is reached for the second time. 

Fig. 47C illustrates PiP playback executed in the case 
where the playback section of the secondary video is 
located at some point subsequent to the end point of the 
playback section of the primary video. In this case, the 
last picture of the primary video is continually presented 
until playback of the secondary video ends. Alternatively, 
it is applicable to end playback of the secondary video 
at the time when playback of the primary video ends. 
[0202] (H) In the case where the current playback point 
of the secondary video Is not changed at relatively short 
time intervals, the PlayList information needs to define 
many pieces of timing information and position informa- 
tion. In this case, it is preferable that the PlayList infor- 
mation additionally Includes User Private Information and 
PLMark information compliant with the BD-RE specifica- 
tions and defines the PiP_position and PiP_size in the 
additionally provided information. Then, the scaler 57 and 



(E) According to the above embodiments, each dig- 
ital stream Is an AV clip compliant with the BD-ROM 50 
standard. However, the digital stream may be a VOB 
(Video Object) compliant with the DVD-Video stand- 
ard or the DVD-Video Recording standard. A VOB 
is a program stream compliant with the ISO/lEC 
1 381 8-1 standard, and Is obtained by multiplexing a 55 
video stream and an audio stream. Alternatively, the 
video stream in anAV clipmaybe compliant with 
MPEG4 or WMV. Furthermore, the audio stream in 
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the compositor 58 can perform the resizing process and 
the composition process according to the user private 
information and the PLMark information. 
[0203] 

5 

(I) It is applicable to present the primary video and 
the secondary video on two separate screens rather 
than one and the same screen. As described above, 
the main video is a portion of the Main Clip specified 
by the MainPath information in the PlayList informa- io 
tion, whereas the secondary video is a portion of the 
SubClip specified by the SubPlayltem also in the 
PlayList information. In addition, it is applicable to 
present the primary video with SD images and the 
secondary video with the HD images. 15 
(J) According to the second embodiment, the Main 
Clip containing the primary video is supplied in from 
of BD-ROM, and the SubClip containing the second- 
ary video is supplied to be stored on the local storage 
200. Yet, it is applicable that the SubClip is sorted 20 
on the BD-ROM and supplied to the playback device 
together with the Main Clip. 

[0204] In addition, it is applicable that the secondary 
video stream is multiplexed with the primary video stream 25 
to constitute one AV clip. 

Industrial Applicability 

[0205] The recording medium and the playback device so 
according to the present invention may be employed for 
personal use in a home theater system, for example. Yet, 
the present invention may be manufactured in volume in 
accordance with the internal structures disclosed in the 
above embodiments. Thus, the recording medium and 35 
the playback device of the present invention can by in- 
dustrially manufactured or used on an industrial scale. 
In view of the above, the recording media and the play- 
back devices of the present invention have industrial ap- 
plicability. 40 

Claims 

1. A recording medium having playlist information re- 
corded thereon, wherein 

the playlist information defines a playback section of 
each of a plurality of digital streams, and includes 
main-path information and sub-path information, 
the main-path information designates one of the dig- 50 
ital streams as a main stream, and defines a portion 
of the main stream as a primary playback section, 
the sub-path information designates another one of 
the digital streams as a substream, and defines a 
portion of the substream as a secondary playback 55 
section that is to be synchronously played back with 
the primary playback section, 
the recording medium further has recorded thereon 



the one of the plurality of digital streams designated 
as the substream, together with an entry map, and 
the entry map indicates a plurality of entry points on 
the substream in one-to-one correspondence with a 

pluralityof entry times on atimeline of the substream. 

2. The recording medium according to Claim 1 , wherein 
the main-path information and the sub-path informa- 
tion includes, for each playback section, timing in- 
formation indicating a playback start time and a play- 
back end time of the playback section on a timeline 
of a respective digital stream, 

the sub-path information further includes synchro- 
nous information, and 

a synchronization point for the synchronous play- 
back is indicated on the timeline of the primary play- 
back section by timing information included in the 
synchronous information. 

3. The recording medium according to Claim 2, wherein 
the entry map is of a first type or a second type, 
the first type entry map indicates the plurality of entry 
times at a constant time interval on the timeline of 
the substream or the plurality of entry points at a 
constant data interval on the substream, 

the second type entry map indicates the entry points 
each located at a head of a complete data set, and 
the entry map includes a flag indicating whether the 
entry map is of the first type or the second type. 

4. The recording medium according to Claim 1 , wherein 
the main stream carries primary video, and 

the substream carries secondary video. 

5. The recording medium according to Claim 4, wherein 
the playlist information instructs a playback device 
to present playback of the primary video and play- 
back of the secondary video on a same screen. 

6. The recording medium according to Claim 5, wherein 
the sub-path information includes synchronous in- 
formation, 

a synchronization point for the synchronous play- 
back is indicated by timing information included in 
the synchronous information, and 
if the timing information is set to an undefined value, 
the synchronization point is set by the playback de- 
vice to a time at which a predetermined user opera- 
tion is received. 

7. The recording medium according to Claim 5, wherein 
the synchronous information further includes posi- 
tion information and size information, 

the position information indicates a display position 
of the secondary video playback when presented on 
the same screen as the primary video playback, and 
the size information indicates a height and a width 
of the secondary video playback when presented on 
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the same screen as the primary video playback. 

8. A playback device for executing trick play of a main 
stream and asubstream, a portion of the main stream 
being defined as a primary playback section, a por- 5 
tion of the substream being defined as a secondary 
playback section, and a start point of the trick play 
being defined on a timeline of the primary playback 
section, the playback device comprising: 

10 

a first conversion unit operable to convert the 
start point into a corresponding address on the 
main stream; 

a second conversion unit operable to convert 
the start point into a corresponding point defined ^5 
on a timeline of the secondary playback section, 
and to further convert the corresponding point 
into a corresponding address on the substream; 
a reading unit operable to read the main stream 
and substream starting from the respective ad- 20 
dresses obtained by the first and second con- 
version units; and 

a playback unit operable to play back the main 
stream and the substream read by the reading 
unit, wherein 25 
the primary and secondary playback sections 
are defined by playlist information, 
the playlist information includes synchronous in- 
formation, 

the synchronous information includes timing in- so 
formation indicating, on the timeline of the pri- 
mary playback section, a synchronization point 
for starting synchronous playback of the sec- 
ondary playback section with the primary play- 
back section, 35 
the substream is associated with an entry map, 
the second conversion unit is operable to use 
the synchronous information to perform the con- 
version into the corresponding pointon the time- 
line of the secondary playback section, and ^0 
the second conversion unit is operable to use 
the entry map associated with the substream to 
perform the conversion into the corresponding 
address on the substream. 

45 

9. The playback device according to Claim 8, wherein 
the entry map includes a flag, 

the flag is information indicating whether the entry 
map is of a first type or a second type, 
when the flag indicates the first type, the second con- 50 
version unit is operable to obtain the corresponding 
address by conducing a stream analysis starting 
from one of a plurality of entry times or of a plurality 
of entry points that is near a requested start point for 
trick play, the plurality of entry times being at a con- 55 
stant time interval on the timeline of the substream, 
and the plurality of entry points being at a constant 
data interval on the substream, and 
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when the flag indicates the second type, the second 
conversion unit is operable to obtain, as the corre- 
sponding address, an entry point that corresponds 
to an entry time near the requested start point, from 

among entry points each located at a head of a com- 
plete data set. 

10. The playback device according to Claim 8, wherein 
the main stream is a digital stream carrying primary 

video, and 

the substream is a digital stream carrying secondary 
video, 

the playback device further comprises: 

a first decoder operable to decode the main 
stream to obtain the primary video; and 
a second decoder operable to decode the sub- 
stream to obtain the secondary video. 

11. The playback device according to Claim 10, further 
comprising: 

acomposite unit operable to present, on a same 
screen, playback of the primary video obtained 
by the first decoder and the secondary video ob- 
tained by the second decoder, by overlaying the 
primary video with the secondary video. 

12. The playback device according to Claim 1 1 , wherein 
the synchronous information includes timing infor- 
mation set to an undefined value to indicate the syn- 
chronization point, 

the undefined value instructs the playback device to 
set, as the synchronization point, a time at which a 
predetermined user operation is received during 
playback of the primary playback section, and 
the playback device further comprises: 

a setting unit operable, if the timing information 
included in the synchronous information is set 
to the undefined value, to receive a rock opera- 
tion specifying the synchronization point under 
a state where the primary playback section is 
solely played back, and the setting unit is further 
operable to overwrite the undefined value with 
a value indicating the point specified by the rock 
operation. 

13. The playback device according to Claim 12, wherein 
the rock operation is made by a user by selecting 
one of button images presented on the main stream 

playback, and 

the composite unit is operable to overlay playback 
of the substream on the selected button image pre- 
sented on the main stream playback. 

14. The playback device according to Claim 1 1 , wherein 
the synchronous information further includes posi- 
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tion information, 
tlie position information indicates a display position 
of the secondary video playback when presented on 
the same screen as the primary video playback, and 
the composite unit is operable to present the sec- 5 
ondary video in accordance with the position infor- 
mation. 

15. The playback device according to Claim 14, further 
comprising: 

a scaler operable to reduce or enlarge an image 
size of playback of the secondary video obtained 

by the second decoder, wherein 

the synchronous information further includes ^5 

size information, 

the size information indicates a height and a 
width of the secondary video playback when pre- 
sented on the same screen with the primary vid- 
eo playback, and 20 
the scaler is operable to perform the image size 
reduction or enlargement according to the size 
information. 

16. A program for causing a computer to execute trick 25 
play of a main stream and a substream, a portion of 

the main streambeing defined as a primary playback 
section, a portion of the substream being defined as 
a secondary playback section, and a start point of 
the trick play being defined on a timeline of the pri- so 
mary playback section, the program comprising 
code operable to cause the computer to perform: 

a first conversion step of converting the start 
point into a corresponding address on the main 35 
stream; 

a second conversion step of converting the start 
point into a corresponding point defined on a 
timeline of the secondary playback section, and 
further converting the corresponding point into 
a corresponding address on the substream; 
a reading step of reading the main stream and 
substream starting from the respective address- 
es obtained in the first and second conversion 
steps; and ^5 
a playback step of playing back the main stream 
and the substream read in the reading step, 
wherein 

the primary and secondary playback sections 
are defined by playlist information, 50 
the playlist information includes synchronous in- 
formation, 

the synchronous information includes timing in- 
formation indicating, on the timeline of the pri- 
mary playback section, a synchronization point 55 
for starting synchronous playback of the sec- 
ondary playback section with the primary play- 
back section. 
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the substream is associated with an entry map, 
in the second conversion step, the synchronous 
information is used to perform the conversion 
into the corresponding point on the timeline of 
the secondary playback section, and 
in the second conversion step, the entry map 
associated with the substream is used to per- 
form the conversion into the corresponding ad- 
dress on the substream. 

1 7. A playback method for executing trick play of a main 
stream and a substream, a portion of the main stream 
being defined as a primary playback section, a por- 
tion of the substream being defined as a secondary 
playback section, and a start point of the trickplay- 
being defined on a timeline of the primary playback 
section, the playback method comprising: 

a first conversion step of converting the start 
point into a corresponding address on the main 
stream; 

a second conversion step of converting the start 
point into a corresponding point defined on a 
timeline of the secondary playback section, and 
further converting the corresponding point into 
a corresponding address on the substream; 
a reading step of reading the main stream and 
substream starting from the respective address- 
es obtained in the first and second conversion 
steps; and 

a playback step of playing back the main stream 
and the substream read in the reading step, 
wherein 

the primary and secondary playback sections 
are defined by playlist information, 
the playlist information includes synchronous in- 
formation, 

the synchronous information includes timing in- 
formation indicating, on the timeline of the pri- 
mary playback section, a synchronization point 
for starting synchronous playback of the sec- 
ondary playback section with the primary play- 
back section, 

the substream is associated with an entry map, 
in the second conversion step, the synchronous 
information is used to perform the conversion 
into the corresponding point on the timeline of 
the secondary playback section, and 
in the second conversion step, the entry map 
associated with the substream is used to per- 
form the conversion into the corresponding ad- 
dress on the substream. 



EP 1 715 686 A1 



28 



EP 1 715 686 A1 




29 



EP 1 715 686 A1 



c5 




— o 
^ —J 



K .2 
u 



Q. 
< 



> ^ 



CD 



^ CO 
CO 




CT5 
CD 

< 
CD 

E 

wwh3 



o 

t 

o 

GQ 




30 



EP 1 715 686 A1 




CD 



31 



EP 1 715 686 A1 




to 



CD t 
> ^ 



-J 
CsJ 



CD 
u 

CO 
CO 



>✓ 
u 

CO 
Q. 

00 

CM 
CO 



> "O 

CU CD 

< ID 




CO CO 



Lo LLi Ed 



32 



EP 1 715 686 A1 




33 



EP 1 715 686 A1 



FIG. 6 A 

IDR Picture in MPEG4-AVC 



I Slice 



I Slice 



I Slice 



I Slice 



FIG. 6B 



Non-IDR I Picture 
B Slice 



P Slice 



I Slice 
I Slice 



FIG. 6C 




Presentation 
Order 



34 



EP 1 715 686 A1 




35 



EP 1 715 686 A1 





CD 


cu 




Q. 




>^ 






1 






l: 


O 




CT3 


o 


O 


o 




CD 


"a. 






1 

CO 


CU 
























_Lo 




iqh 






me* 


UJ 


^1 


1 


! 






o 


LU 


LU 




1 


1 




to 












00 



CD "O 
CX3 § 

CT 4-' 
03 C 

I O 
.$5 CL 



.2 o 

4^ I 

'cn i 
o tx 

D. LU 



—I o 



o :5 



LU 

i 

CO 



1 1 1 



o 

LU H 
GO ^ 



CD 

Q. 

>> 

O 



A 7 




LO 

u 



"7^ 



J 5: 

^\ ^ 

S- CD 

C 4-J' 

^ ! I 

Q» CD 

UUI J3 

O 

CO 



























o 




a 






















• • • 






• • • 














O 


1 






1 






CL 




1 














LU 




^1 












LU 



u 

N 
Nl 
N 
N 
N 





















CD 
O 




C 

CD 


D 


Sequ 



o 

CO 

o 







































cn 












1 

CD 
















</) 










































































































































TO 








i 


Ne-1 


"O 












o 


o 








Ne 


cu 

1 






i 

CD 






















start 


O 










CD 


CD 


1 

O 

1 








rie 




PID 




f t 










































C 


o 


1 




I 






1 


i 

O 


• > • 




Ne-1] 


igh_e 


CD 

I 


ne_stream_P 


itream 


• • • 


tream 








mmmJ 


to 

1 






CD 






1 




1 


o 


typ 


.EP_ 


.EP_ 


one. 




CD 

t t 


CL 


EP_ 




one 




r^one 


o 


am_ 


1 


1 


J 






am_ 


! 






1 

Urn. 






o 

1 


o 

1 










1 


o 

1 

co 
E 

i 




P-^mapJo 


'eam„ 


CD 

(/) 


ruber 


mber 


_map_ 




*eam_ 


_stre 


mber 


mber 


Q. 








E 


E 








r\ 


E 










(/) 












CO 


UJ 






LU 




LU 



























36 



EP 1 715 686 A1 



DO 



rrs 



CO 

_ 



CD 



CD 



t/) 
CP 



03 



in 



u 

CD 
CO 



to 



C5 



i/i 
CD 



O 
CD 
C/) 



CT3 



U5 

to 

CD 



O 
CD 
CO 



CX5 

+-» CD 

(/> «^ 

CO 

CD 



<— o 

■4— » CP 
CO ^ 

to 
CD 



CM 



o 
to 



to 

CO 



_ 


<D 






Lev 


imei 










CM 




O 















II 



o 



Cmm! 
LU 









cz 


II 


(i 


"4 


1 1 






TO 


CC 




■1—' 


to 






1 

Q 


yj 


1 i 1 


I 


{ 










85 



to 
II 

'o 



LLl 



CO 
li 

4..J 

c 
'o 

Q_ 
>^ 

c 

1 1 i 



II 



o 

1 1 I 



LO 


LD 






If 


II 


t 






CT3 






1 


1 




CL 


LU 

1 






1 




Q- 




t/0 



CO 


CO 






II 


4-' 






CT5 






tn 






t 






LU 


i 1 1 


1 

























c 


If 


II 


4^ 


„l J 














(-0 


CO 


1 


1 


ex. 




LU 

1 


LU 












00 



cu CC 
> JZ 

_i ^ I 

Q_ 

"P LU 

CO 



II 



o 



l5 



tD 


to 


-| 1 




II 


II 










03 








1 


CO 




i 

CL. 


LU 


LU 


1 






^1 




0 


c/> 



Jl^ 
*o 

n 



t5 











II 




■M 






Ct5 


CD 


CO 


So 


^1 


1 






t 1 1 


LU 


1 














CO 



CM 

II 



o 

^^^^ 



II 



CO 



! 



CM 
C 
11 

as 

CO 

I 

I I I 

I 

Q- 



C 

II 

a. 

CO 



a. 

CO 



LD 
CZ 



CO 



li 



Q. 
CO 



CO 

c 
85 



CM 



CO 



-Ah,^ 11 



CD 



<<r3 CD 




I ^ 
-J o 



<D 

3 CD 
10 



C C/) CD 
j 0) ^ 



CD 



CO CD 
3 CO Q 

I ^ I 



cz CO 

Mnil| ^^^^ 

I ^ 

-J ^ 



CD 

O 



CD 



C CO CD 
I CD '-^i 

< c 



C (/) CD 
3 CO j •4-' 

.1 g.Wc 

— -J fj iZ m 
< <: Z> CD 

: J 



<D 

> ^ 
CO 



37 



EP 1 715 686 A1 



O 



I 

o 



I 

o 



UJ 



CO cr 

CO 

CO CO 

o !<: 

E oo 

cj> o in 



CO 

^ CO* 
O I 5C CNJ 

o 2l i£ *^ 
00 I li 



I 



a> — I ' 

y o S o 

cr ^ o 

v.— I ! 

CD CO UUl 

> I II 



2j ^ 



CO X 



1 1 1 ^ 



n 
UJ 

o . 



o + 

CD — 



+ 

I I 

00 ir 



UJ -Q 



o 1^ 

§ CM 

_j 
0) -^j< 



7 



o 



I 



II 



UJ ^ 
Z CD 

a. CO 

CO ^ 



CM 

c 

ii 

—J .± 



UJ O 

a. CO 

CO -J 



CO 



i Hh 

^! CO 
Z GO 
Q- C/) 
CO -J 



o 



II 

CO 



UJ 

I CO 

21! no 

CO 

CO ^ 



o 



I 

CO 



5 
o 

I 

UJ 



S5 



+ 



o 

UJ ^ 

Z 
CO 



O 

r\ ^ 
CO 



o 
-J 



S5 



5 
o 

Z 

§5 



UJ 



II 



CM 
II 



o 



+ 



I 



CXm 

UJ o 



C/) CQ 



UJ O 
I CO 

I— CO 



CO 

11 

^ CM 
O ^ 

LU O 

I (O 
CO CQ 

Q I 



II 

n ^— 
UJ O 
I (O 
CO CD 
h- CO 



o 



& CP 

I 

CO 
I — 



o 
_J 

I 

co' 



+ 



5 
O 

UJ^ 
00 

a. 



5 

o 

UJ ^""^ 

1 

CO 



o 



111 ~r 



O 



CM 



f I I + 



5 T 



CD 

c 

05 
O 

I 



I 

CO 



O 

i + 

CD 

c: 

U 

I 

a; 

03 

I 

CO 



■5 A 

Q-CM 
CD ^ 

c 
cn 

o 

j 

O) 
CO 

I 

(O 



r- O 
O-CT) 

cu ^ 

C 
CO 

a 

! 

CD 
CU 

1 

(O 



c O 

o II 

I 

CD 

u 



o A 



CO 

I 

CO 



c 

CO 

o 

i 

O) 
CO 



o A 

Q-CM 

i 4. 

Ciiiip 
CO 

o 



'5 A 

O) 
CO 

o 



.E II 

CO 

a 



CO 



CO 

I 

CO 



D> 
CO 

I 

CO 



CO 

I 

CO 



4-^ O 

II 

Q.CM 

CO 
O 

I 



05 

I 

CO 



38 



EP 1 715 686 A1 



CX5 
CD 



CD 

O) 

TO 

O 

15 
a 
o 



— i C 
— * J— 

o 



.0- 
D 

CO 

o 

cz 
"a. 

• wmmm 

o 



<D 


cy 


» 


05 


TO 


o 


(n 




c 




a 


0) 


OS 


o 




03 


CO 


1 

CD 


5 sec 


Some 


E 


"5 




o 


5 


c 


CO 





E 

4-' P CD 
_ ^ CO 4-J 

o>< 

o 



X5 5 
CD- , 

E 4-''< bE c 

O 3 ^ CO CO ^ 

CL ^ ^ 



o 

JD 
CO 



CD 

CO ^ 4-^ ^ 

CO 

U CO 

L_ CD CI? ^ rn 

Q-COQ-iS 



o 

to 



o 



o 
o 



d 
CO 



E 

CM 

o 
o 
o 
o 



u 





'a. 




c> 






CO 


O 


o 


o 


o 


o 


o 


o 


O 



a, 
a 

o 
o 
o 
o 



=1 



a 

LO 

O 

o 
o 
o 



c 








in 




</) 


(/) 


to 










+-» 






CNJ 


CM 


CNJ 


E 










oJ 




CO 




LO 


o 




O 


o 


o 


o 




O 


o 


o 


o 




o 


o 


o 


o 




o 


o 


o 



i 
i 



— CD 
> CO 

^ .2 



CO 

u 



CL 
< 



3 

o 

I 

T3 
CO 
CD 



CO 
CD 



CD 
(A 

CO 

LZ 



> CO 

E 

CNJ 

CO 

J) 

iZ 



CO 
CD 

< 

E 
"o 

> 



"T CO 
ID CD 
CO J= 



CD 



CD 
> 



-r-«t CO 

if y 

CO (/) 



39 



EP 1 715 686 A1 



CM 
LL. 



CD CI CP 

TO oo i5 

-jni -O I — cs 
q3 ci- 

13 a£ g ^ LU 



o ^ 



<i> c: 

J— CO .h; oo CJ 

1^^^ «4'<"^ '^^^J |u)lLlllllLII C^CS 

O CD 

^ to >^ CO 

^ CD ^ *-*-J 



CD 



w »— CD . . 

E .2 t;^ c: CD 

tZ CO CO O 

Li— "O j — CO 

Cl 3 f iii m ii f ^ 

rp r— 

g § OJ ^ 

= "ir3 CO 

E .Sii CO 

ul. -jr: "O I— CO 

^ _ 

q: ^ ^ 



CD •-= 

to m >^ ^ 
^ CD ^ (Q UJ 



CD CZ CD 

E 

CO 



_ CO 

CO .S=i CO o 

♦zi "o H — CO 

CD ••— ' ' 

to CD ^ 

CD f=- _Q LU 

CZ f> | 



CD 



E .2 ^ £ ^ 

o S .E CL. 

^ CO CD ^ 



Si S CD +^ 

>— O 4-^ j3 CD 

CO •*=! — 

Li_ <LJi j— CO 

O CD ^ ^ 

^ ^ CD 

•—5 CD r— CT> UU 

<C £~ 



CD C CD ^ 

E .2 c: CD 

fcZ CO CO O 

-Jsi T3 I — CO 

.0 a, Cl- 

^ CD 5=i _Q LU 

<C d. p: 



I 3 

CO .•== 



^ ^ CD 
CD -z= O 

c5 H 



a 

CO 
UJ 



u 

UJ 



u 

CO 

LaXj 



CD 

O 
Ct3 

CO 
UJ 



CD 

O 

Ol 
CO 

Q. 



CD 
UJ 



CD 

O 

03 

CO 
UJ 



CD 
U 

CO 







PTS 


\ 

/ 


f 


PTS 


\ 
/ 


f 


PTS 


/ 




PTS 






PTS 




(- 


PTS 


\ 
/ 




PTS 






Sid 



, . CL c^ 

mumm 

-J 03 



CO 



CO 

Cl. 



■ 



/IK 



f 



"05 ^ 

CD 
mJ 

+-J O 



40 



EP 1 715 686 A1 



CO 

a 

1 1 




CO 
CD 



3 
I/O 



I Z3 r- 

C 



CO 



CO 



C/5 

to 



CO 



CO 
Q 

o 



CO 



<0 

> 

CO 



o 

CO 



CD 

o 

CO 



0) 

CO a 

CD 

CO o 

UJ CO 



<D 

CO u 
UJ CO 

CO u 
UJ CO 

+-» 
CO u 

UJ CO 



<D 

CO U 
UJ CO 

Q. a. 



1 1 I 



J£3 U2 



CO 
Q 

o 



CO CO 



CD 

CO u 

UJ CO 
CL CL. 

CO U 
UJ CO 

CL a. 



CD 

CO o 

UJ CO 
□- Ql. 

0) 

CO u 
UJ ^ 



CD 

CO u 

UJ CO 

cl d 

CD 

CO u 

UJ CO 
CL CL 



CO 
O 

o 



CO 



CD Co- 



co 

o 



CO 
Q 



CO CO 

C3 



CL 



OO CO 



21! 



CO CO 
I — I 

(Z3 Ci_ 



CO 
O 
O 



<>0 CO 
I — 



CO 
Q 

o 



CO CO 

I — I — 
CD ex. 



CO 



CO CO 

O 



in 

o 



OO CO 



CO 



CO CO 
f ""' 



^ b ^ 

^ CO Q_ 

^ UJ 



CO 

CO .2 
o ^ 

O O" 

UJ <C 

^ 



13 

5 CO 
o _ 

o £ 

UJ z: 



c 

CO 



o 

t: qi 

GO .2 

^ CO 

o cr 

CL CJ 
UJ < 



CD 



,l 1 

c 
o 



CD 
CO 



R o 

ULi IZI 



CO 

Q 



41 



EP 1 715 686 A1 



Q 

•O 



+ 

00 

Q 



CO 

o 

00 

o 



Q 

oo 
o 

CO 

I 



00 
Q 



K Q 



-- a. O) 

CO 

c5 



CO |_ 



CD 

OO O 

00 U 
UJ 03 



CO ij 



CO u 

CO o 
LU 03 



00 o 

yj cu 
a. Q- 



CD 

CO U 
UJ ^ 

cu 

00 CJ 
LU CtS 

Q- a. 



CD 

CO O 

LU Ct5 



CD 

CO u 

LU cu 



CD 

CO o 
1 1 I fu 

4-' 

CD 

LU CU 



> 

CD 



OO CO 



CO 
O 
O 



CO OO 

I — I — 

CD cu 



cyO 

O 



OO OO 



CO 

o 

o 



OO OO 
Q fez 



OO 
Q 



OO cyo 



CO 
CJ 



OO CO 

tr I— 



3 



O 

n 



CU 
CO 

o 
o 

yj 

V 



o 
o 



CD 
</) 
CU 

O 



g 

=5 o E 

CT O 

O CL O 

<C LU 2 



CO 



OO CO 



Q 

O 



OO CO 



CO 
Q 
O 



CO CO 



O 



OO OO 



CO CO 



CO 
CJ) 



CO CO 



CD 



CU 
CO 



u 
o 



o 
o 



O 

a. 

c: 
o 

Z5 CJ 

?T o 

CI 



CD 
CU 

15 
E 

o 



c 

CO 

o 



C/3 



CD 



CD ^ 

CO 



42 



EP 1 715 686 A1 




O 

CO 
OS 

E 

« ■ 

CO 



s 

\ 

\ 



o 

o 
u 

cn 



CL 
TO 

O 



03 



a; 

CO 
CL 



in 

cl 
cn 

CP 

CD 
> 



U 
CO 

B 















1 


r— — 1 




Lo 




lah 






1 




1 


o 




UJ 




1 

00 


1 






05 




CL 







• » • 





t 

O 



UJ 



u] 



• • « 



o 



03 
CD 



CO 



/ rvi 

O 



A" 



CD 



C\5 
CD 

C/> 

i 

CL 

iwiiLn] 



o 



C/) 
CD 



CD 

i 

CL 

\ I 1 



O 



o 



,CD 

c 

I 

o 



UJ 



C 



o 



"IS 
CO 



to 

o' 

CO 
CD 



to 

o 



UJ 



J 

CD 



Ct5 
CD 

■I— » 
CO 



I 

CD 



CD 

Cl 

>! 

I 

^^^^ 

cu 
to 

i 

CL 

I I t 



t 

CD 



cn 

CD 



I 



CL 
LU 

I 

CD 



I 

CD 



cn 

CD 

c 

CD 

1 

o 
-J 

^1 
1 1 1 



CD 



t 

CD 



CO 

to 
CD 

ro 
to 

I 

Q 

E 

CT3 

CP 

to 

I 

CD 
d 
O 



CL 



CL 

\ , y „, i 



I 



o 

Q 

CO 

1 

CD 
C 

o 



CO 

1 

2j 



■ 

N 
N 
N 
N 
N 













c 
















CD 








O 


E 






Sequen 


03 


CJ 




Prog 





JO 

! 

Sj 



77 



t 

O 

! 

CX5 
CD 



I 

i 

CD 

o 



o 

i 

CL 



43 



EP 1 715 686 A1 



g 

OS 



cn 
o 

a; 
(/) 
to 



CD 

4-' 

o 

ifi 

> ^ 

tn 

o . 



5 



o 

5 



a. cu 



CO 



cu 
I/) 



o 
a 

cu 
> 

o 

CD 
CD 



in 
I £ ^ 

LU g 
•z CO 

o 



CD 
_Q 

_ 
00 



CO .== 

>^ to 

CD 



CD 

CL 

in 

4—' CO 
CZ CI 

o o 

u CO 



co 



o 



o 

I 

a. 



cu 



o 



CD 



CD 



CD 



CD 



CD 
ii 



CO 

I 

a. 

LU 



CO 



II 

C/5 

I 

CO 

I— 





LO 


d 




n 


II 


4-' 




Lmmm 






C3 






CO 


CO 


I 


^1 


LU 


i y 1 

i 


I 


CO 


o. 


1— 


CO 



LU 



CO 
c 

il 



CO 

oo 

I 

i 

CO 



LU 











II 








i— 




CO 


Ct5 






CO 


1 


^1 


LU 


LU^ 


1 


CO 






CO 


n_ 







CO 

II 



cc 
CO 

CL-' 
t 1 I 

! 

CO 

n. 











II 










CO 


CO 


CO 




! 




LU 


I 


^1 




Q- 






CL. 













II 


-1— » 








CO 


CO 


4—' 




CO 


1 


i 


CL. 


LU 


It 1 

1 


1 


CO 






CO 





<U CO 
*o 1 1 I 



LD 



Q_ CP 

I :3 53 

I 

C^ 



44 



EP 1 715 686 A1 



CD 

CD 

CO ^ 

-2 E 



CO O CD 

5 2: c/^ 

O Q- CO 



CD 

CD 

CD d 

O CD 

i5 2: CO 

' } •4—' 

O ^ 
CD "rS 

C^ X 
O. (D O 



85 



II 

n 
CO 



CD 



CO 



CD 

— ' 

CO 

r~ CO 



CO 

<D 



CO 
CO 



o 
o 



CO 

> 

CD 

CD 
CO 

O 

c 

CO 



o 
u 



LO 

II 



*o 

&HMM 

4-» 

c 

UJ 



O 

a. 



LD 


LO 








!i 


c5 


CO 






00 

1 








CL 




^! 

CO 


1 






CO 















il 




,| ,„ t 






CO 


CO 










1 




LiJ 


UJ 




1 




CO 






u5 






"53 


CO 


> 












cz 


C\J 





(0 

> 
o3 



CD 

• «II|||W 

CO 

CO 

c 

o 
o 



1\ 



CO 

CD 

CD 



CO 
•M 
t/) 

o 
o 



CO 



45 



EP 1 715 686 A1 



O 

a 



o 

4-» 

(/) 

CO 

■ MM* 

Q- 

I 

03 



I 

05 
O 

tn 



! 

"■M 
to" 




0) 

4-' (/> 

CD CQ 
C/3 O 

^ 2 

.§■1,1 
c 2 >. 

o a.^ 

GO 5^ 



il 

n 
CO 



05 
> 

03 
05 

o 

d 

03 

4-* 
</) 

c 

o 

u 



03 
> 

c 

CO 

+-» 
Q 

4-' 

03 

c 

o 
u 



c 

II 

a. 

CO 



> O) 

■hi £^ 

CO 

»— CO 



LO 

I! 



O 
C 



LO 
C 

CO 

^! 

LO 



LO 

+-» 
CO 

I 

Cl 

I 



I 




03 
> 

a3 



CD 

E 

d 
03 



to 

o 



LO 

II 

00 



05 
> 

CP 



cu 



03 
CO 

O 
u 



II 

CO 



I 



a> 



«iMiiMaij| 

z:i oo 



CO 



46 



EP 1 715 686 A1 




47 



EP 1 715 686 A1 




d o 

03 

S a> iS ^ 

^ 1 j~ i^_> 

"O -3 CD Q 

c ST? 

rvj cXm ^ 



48 



EP 1 715 686 A1 



CVJ 

C2 



E 



ID 




~l 


Q- 


am 


3 


St 




1 






E 




o 




1 


ark 


CD 


E 



1^ 



A 




• • • 



o 
E 



CO 



CO 



49 



EP 1 715 686 A1 



o 



c5 



CO 



CNJ 

CD 


















If 








TO 




4— » 


CO 


CO^ 


&' 


]q 1 


CO 


1 









X5 

1 


E 


E 


CC 






„Playl 


.time„ 


o 


1 


1 




CD 

























, ■ 


CO 


CO 




i 






1 














CO 







•4—' 




Jl 


It 

4 — > 








CO 


OS 


CO 


Jo 


1 




CL. 




yu 

1 


LXJ 

1 


CO 




t— 






CO 



CO 


oo 






11 




in! 








CO 


■♦— ' 


OO 


1 


i 


UJ 




1 




CO 




1— 






oo 









c: 


In 


tt 




■*-» 


CO 


res 


CO 


CO 


1 




CL. 




UJ 

1 






CO 








LO 



CD 



II 

O 



> it 



CZ 

II 



CO 



II 

cy^ 



to 



^5 



CO 



II 

C/5 



V V ^ ^ 



50 



EP 1 715 686 A1 



O 

c 



o 



r 



If 
ii 

W ^ = Q. 

V_ I .1 CH) 

u. ^ I 

__A 



I 

o 



to 

CD 

' "ji ™ ^ 

Wmmmm 

CO -D 

5-. CU 

O <^ 



x: CO 

CO £ 

Cl ^ 

CD 

CO o 



f— w 

<u 5 

CL CO 

0 E 

CO jj) 

1 ^ 



CD 



to 



C/) 

E 

CD 

s_ JO 
O CL 

r: >^ 
to 



CO p. 
CD 2^ 



ClO 

CO 

CL'o 
CO CO 

_ CD 



CO 



CD 
</) 
CD 



c: 

CD 



CL ro" 



CO 



o ^ 



CO >^ 

Cl CO 

•B >^ 

03 

d fl) 
CD 

to 

t/) CO 

O CD 
•~ ^ 

Cl to 



CD 
> 



u 

CO 

a3 



CO 
CD 



Cl CJ) 

to CD 

^ Cl. 

<C CO 



CO 



o 



CO 



CD 

CD 
CL 

o 

Z5 
CO 



CO 
C 

o 
o 

<D 
CO 



CO 



CD 

Cl 



CO 
CO 



CNJ 

I 



/ 



CD 

IE 

CO 

^! 

CD 



E 





















CD 




■4-» 


E 






E 


^1 


1 


P Pla 


! 




E 




Ou 


<D 


o 

1 


tern. 


i 


ayl 


QL 


CD 




^1 






i 


CO 


>t 






ubP 


CO 


c 


CO 




>^ 
CO 


CO 

1 


CO 


Su 




Sync_ 



a; 

SI 
CO 




CO 

•mmmJ 

CL 



'co 



CO 



.<>2 

JCO 

a. 



4-* 

CO 
n 

3 
CO 



o 



51 



EP 1 715 686 A1 




52 



EP 1 715 686 A1 



LO 

c 

il 

CO 



II 



n 
CO 



CO 

c 

H 
CL. 



cr 

II 

CO 



It 



CP ci> 

CP o 

— i CO 

GL> 



LO 
11 



o 
c 

UJ 



II 

E 
o 

CL. 



LO 
LZ 
il 


LO 

1 


CO 


iP_Staii 


1 

CL. 


^ t 1 

1 

85 


III! 

1 

CO 
\^ 



CO 

II 



!i 


1 J 

II 






CO 
UJ 


CO 


PTS_EP_ 


1 

CO 



o 

LU 



il 



CO 
t! 



00 



85 



O 

>. 
u. 

+-» 

c 

iXJ 



tl 




TO 


c5 


CO 


CO 

! 


^1 
UJ 


CL. 

LU^ 

CO 


1 

CO 



II 



o 



I M 



Ct5 
CO 

UJ 



CL. 
CO 















il 




ti! 


t:! 


03 


TO 


co^ 


CO 


UJ 


UJ 













1 E & 

c 

*X3 y 

















CCS 


TO 


00 


CO 


I 


1 

ry 


1 i i 


LU 


1 




CO 


^1 








85 







1^ 


ii 


II 

1 ■ 


s 


L_ 


Lmmm 




TO 
4—' 


TO 


ni 


CO 

i 


Jo 

1 


a. 


CL. 






LU 

1 


1 




CO 






dmrn 






85 



LO 

CM 
■ 



OsJ 

L_ 

CD 

Q 




CO 

o 



LO 
4—* 

ii 

TO 
CO 

LXJ^ 
CO 



LO 

il 

TO 



CO 

I 

CO 



00 


no 






tl 




4—* 


i 


TO 




Jo 


CO 


i 

LU 


i 

0 
UJ 

1 


CO 






So 



H 
4— » 

TO 
4— > 

I 

UJ^ 

Q_ 
CO 



t 

> II 



^^^^^^^^ 



O , 
CO 

cn 

II 



V 



CO 

00 



Q. 

00 



11 

CL. 

CO 



CO 



uo 

If 

CO 



00 



ro 
85 



* ^ 
a. 

CO 



85 



L« 0) <D 
g CP u 



I CO 

COCO 



53 



EP 1 715 686 A1 



1^ 



re 

> 

CD 



C5 

o 

TO 

O 
O 

IHMMMM # \ 

CO 



CO 
0) 

CO 

Q 
c 

CO 

O 

u 



LO 

II 



o 

n 





LO 


cz 




li 


ii 


1 


1 I 


03 








oo 


CO 


1 


1 






LU 


cu 


I 


1 




CO 






&5 



11 



LU 



CT3 
CO 

I 



CO 



II 

OS 



^1 

I 



<D O) 
CP >-i^ 

O. ' ^ 

in CO 



CD 

e3 



CD £5 

"O LU 



CT5 
> 

CD 



CD 



c 

CO 

o 
o 
c 



03 
> 



CD 



I — 



CO 



c: 
o 
a 




LO 



CD 



E 

CD 



CO 



CP 



A 



a. 
o 



in 



I 



GO 



CO 



QQ 



CO 



I V 



iS CP <u 
^ 

o "cS "cD 

^ .E E 

E E 



CO 



CD CD 

>> >^ 
CO CO 

o n 

- - 



OQ 



CD ^ 

5 I 



o • 



CO 

■ ■ 

> 



CD 

O , 



> 

CD 

■o 












CO 










ii * 














CO 



























TO 







Lart=t1 


ti 

t! 
cts 


CO 


CO 

1 




CO 


CO 



35 



CC5 



CO. 



cn 

"If 
+~> 


Jl 


cts 


i. 

CO 


PTS_EP_S1 


CO 


1 

CL. 

^ ^, 1 

1 

CO 



! CL. 

1 <^ 

I 

I CD 

; oo 



LU 





li 


c5 

4—' 


,i,„ 

CTS 


CO 

1 

UJ 


CO 


1 

UJI 


1 

CO 


85 



-I H 

I 

I Q. 
» CO 

I 
I 
1 

If 

. c-o 

I 
I 

1 eg 



LO 


LO 








11 


4-' 


4-> 






OS 


CC 




+_i 


CO 


CO 


1 








1 


LXJ 


CO 


1 


CXm 







11 

♦ CL. 

1 

I 

I LO 



00 

11 



CO 



CO 

S«NMMt 


CO 

Ii 

TO 


PTS_EP_S1 


CO 


cJ 

LLi 


85 



t Q„ 

' CO 

\ 

1 

' CO 

! CI 

""■""^ ^^ 
Q- 



i » I m il * < ' ' t J 



UJ 



3l 


II 

4--» 


rq 
00 


CO^ 


1 

CL. 
LU 

1 




CO 


CO 


• ■ 



CM 



CO 



85 



CD Q- 

33 ^ ^ 

hbpmJI i 



CO 
0) CD 

CD 

> CD U 



54 



EP 1 715 686 A1 



CD 



I 



CO 
CP 

CO 

I 

i I I 

51 

CJ) 



CO 



CO 

u 

CL 
CT3 

o 

-M 
C 
0) 
c/) 

dI 



05 

> 



o 

03 

a3 



o 

< 

v» 
05 
"O 

o 

CJ 
CU 
CO 



CU 



o 

OB 
.O 

g. 
< 

o 



o 

D 



S 
o 
15 

o o 

CD 

-O 
CO 

o o 



CP 



C;3 



o 



CO 0^ 



03 



CO 



CD 



CD 
(/> 

CO 

03 
CD 





to 


o 


o 










TO 




^^ll^ ^^^^ 




CO 










m m 




to 





CD 



CO 
O 



4-> in 

CO ^ 

2 

' _C5 

^^^^^^^^ ^hXm 

^mmm ^ 

*r- 
CD 

<D "O 

Q. CO 

00 =3 

^ E 

O. 03 

\mmm i_ 

^ cn 

g C9 

CT3 O) 



CD 



■4—' 4^ 



CO 



CD 



CD 

cn 

CD 

.y CD 
tf- CO 

CD 
CD — 

c-> JS 
^ c 

**** TO 



C3) 

O 

"D 
13 

CO 

O 



CO 
Ql 

g 

CO 



CD 
</> 

E 

.g 

TO 
r- 

o 
o 

CD 
CO 

CO 



55 



EP 1 715 686 A1 



CD 
CO 
O 
Q. 

E 

o = 



o 



a 

CO 

15 



4-* 
CO 

u H- 

CD ~ 

CD vi-« 

Q O 



J/J 






o 




o 


E 




E 


• 




• 

CM 


o 




o 


o 




o 


o 




o 


o 




o 



• • • 



00 
CM 

(3 





O 

CO 



CL O 
A_ 















u 


a 
■ 


u 


CO 




LO 


o 


O 


o 


o 


O 


o 


O 


o 


o 


O 


o 


o 



r 










cn 




</> 


to 


CO 








1 1 








OvJ 




CM 




CM 




• 






E 


a 




CM 




CO 




in 




O 




o 


o 


o 




O 




O 


o 


o 




O 




O 


o 


o 




o 




o 


o 


o 



i 



56 



EP 1 715 686 A1 



O 

o 
















CO 


■4— » 




on 





















Mai 


nve 




o 







z 



CD 

3 > 

o 
o 



TTTK 



5 



O 03 
CD c 

> in 



.9 

< CO 




.2 >v 
o 

CO 



E 

CD 



CO 

>^ 

o 



CD 
O 



O 



rv CO 



r 



C7> 



I in 
CO "43 



oo 



O CD 



\ —J 



CC CO 

CD o-S £ 



CM 



CO 



X 



IT 



CO 



CD 

> CO 

CD £5 Q- 



CM 




0^ 

o 



o 



CX5 

OO 

i 



0.03 















CD 


LU 






>v 










o 




00 




DO 




\ 






CO 



\ 



^ /• ^ L 

0^ CD 



o cz 

i — (D -zz 



CO 



CM 



v_ CD 3 

<C E S 







13 TO 

^ J n ^ 









3 



O 



CT) 





57 



EP 1 715 686 A1 



FIG. 30 




Jump Playback from Arbitrary 
Point on Playltem Timeline 




SI 



Calculate Offset a indicating Point on Playltem Timeline 



S2 



Convert Offset a to Coordinate on 
MainClip Timeline (ln_time + Offset a ) 



S3 



Convert Coordinate on MainClip Timeline (ln_time 
+ Offset a) to Address a using Main Clip EP„map 




Convert Offset a to Offset /3 of 
Coordinate on SubPlayltem Timeline 
(Offset a - Sync_Start_PTS_of_Playltem) 





^ ^ — 'S6 


Convert Offset /3 to Coordinate on SubClip 
Timeline (SubPlayltemJn_time + Offset fi) 


N 






Convert SubPlayltem_ln_time 
Offset /3 to Address /3 using EP_map 




N 




Simultaneously Read BD-ROM Starting from 
Address a and Local Storage Starting from 
Address /3 



C END ^ 



58 



EP 1 715 686 A1 



FIG. 31 



Closest Entry Point 



Upper 
1 St Level: 
TS Packets 



i 




SPN=nl 



SPN=n2 



SPN=n3 



256kByte 



Upper 
2ncl Level: 
EP_map 



i 



256kByte 



A 



Entry Point=#1 Entry Point=#2 



i-- 

Entry Point=#3 



SPN. 


.EP_ 


.Starts 


=n1 




-EP. 


_Start= 


=t1 





Upper 3rd Level: . 
SubClip Timeline 



SPN. 


-EP. 


.Start =n2 


PTS. 


.EP, 


.Start=t2 





5 sec 



SPN. 




.Start=n3 


PTS. 


-EP. 


.Start=t3 





2e 



5 sec 



V 

SubPlayltem_ln__time 



Sync_Start_Pts_of_Play Item 
Upper 4th Level: SubPlayltem 



v — 

Offset >5 




TM 



Timeline 



Offset >S 



Middle Level: Playltem Timeline 



Lower 3rd Level: 



In time 



r W ^ 

Entry Point=#1 



Lower 
2nd Level: 
EP^map 



PTS. 




3tart=t1 


SPN_ 




_Start=n1 



V 




Offset a 
Offset a 



~ 

Point to Start Playback I 

^ — 



Entry Point=#3 



PTS EP Start=t3 



SPN EP Start=n3 



Entry Point=#2 



V 



PTS_EP_Start=t2 


PTS_EP_Start=t4 


SPN_EP_Start=n2 


SPN_EP_Start=n4 



7 



Entry PDint=#5 



PTS EP Start=t5 



SPN EP Start=n5 



Entry Point=#4 



V 



Entry Point=#6 



PTS. 


_EP. 


.Start=t6 


SPN. 


EP 


.Start=n6 



Lower 
1 St Level: 
TS Packets 



SPN=nl SPN=n2 SPN=n3 SPN=n4 SPN=n5 SPN=n6 











1 




















i 










i 










i 














i 









59 



EP 1 715 686 A1 



FIG. 32 



Conversion 
to Entry Point 



/Conversion of Coordinate TM on SubClipX 
\Tinfieline to Access Point Address /? ^ 

nI/ ^S11 



Sequentially Add Tinne Length of Each 
PTS_EP_High until Total 1 Exceeds 
Coordinate TM 



S 1 2 



k *- PTS_EP_High_id at 
which I Exceeds TM 



I 



SI 3 



Sequentially Add Time Length of Each 
PTS_EP_Low to Total 2 of up to 
PTS_EP_High(k-l ) until Addition 
Result Exceeds TM 



I 



S14 



h *- PTS_EP_Low_id at 
which TM is Exceeded 



V 



I 



SI 5 



Identify Entry Point Closest to TM using k-1 
and h-1 , Calculate SPN Located at TM using 
SPN_EP_High and SPN_EP_Low, Convert 
SPN to Address 



r 



S18 




Start Stream Analysis from 

Converted SPN Address 



Stream / 
Analysis ^ 



NO 



819 

PTS" 

"Exceeding Coordinate 
JM Reached^ 

YES 



Address ft 

Converted SPN Address 



S20 



Address Address of PES 



(return^ 



60 



EP 1 715 686 A1 



CO 



5 
O 



C3^ 



CO 



CO 



I 

to 



5^ 

I 



E 
o 



._ 

CO 

a> 
o 
o 

< 



A 



a3 F 
o 

n C 



CO 



0) 

o 

CO 
II I 

Q. 



o 

mrntnal 

1 

^ I I 

I 

oo 

r-> 



.+-» 



Cl- to 
CO 

Ql. ^ 



to 



O 

cz ^ 
CD O 



^ O ^ CO 

a> "53 o J= c 



C 



MMHlJ C^^*^ 1^2^ 

CO CL 



a> cl 

> LLl 
CD 



CM 



to _J 



P (JO 



61 



EP 1 715 686 A1 



LU 

i 



TO 



I 

to 



o 

TO 
X 



5 



to 

„^ 
CP 

cu 

o 

X 
O UJ 

CO J--- 4-* 
t — ""-< 4—* 

X5 TO — 

. c: ^ 
-XT <u -c: 

TO— ^ ^TO 

^1 E ^1 

{ ^ 

CO +-» CO 
I — O I — 
D- 



• 



i 



5 
o 
-J 

I 



UJ $ 

f o T 
I — j) 



UJ, §: 



E 
c: 



"o 

CD 
U 
U 

< 



to 
o 



A 



+-• 



CD 



CO 



o 
I 

I 



o o 



CO 



C CO 

3 to 
u -£ cu 



CP Cu _ 
0) j TO 

— ^ <^ X 

4«>> U— I 

to cu 



CD ex. 
> yj 



1 



0) 



c5 



5 

o 



I 



"0) Q- Q. 



62 



EP 1 715 686 A1 




EP 1 715 686 A1 



FIG. 36A 



Width 1 920 Pixels 



Primary Video 
(HD Image) 



Width 720 Pixels 



Secondary 

Video 
(SD Image) 



Height 1 080 Pixels 



Height 
480 
Pixels. 



FIG. 36B 









Secondary Video 






(SD lmage)| 






1 1 




Mr 

V Height 
Xl/4 



Height 
X2 



64 



EP 1 715 686 A1 



CD 

03 
O 



52 g 



c 
o 

to 




CO 

■ 

LL. 




o 

rg 
O 
O 

o 
o 




in 

s 

O 

o 
o 
o 



i 
i 



— CD 

> CtJ 



to O 

OS 
O 



O 

i CD 

cu <; 

CD 
-J 



0) 
1/5 

CD 



CD 
> 
CD 



c5 



CD 
>^ 

00 
CD 



CO 
CD 

< 
0) 



CO 
CD 

CO 3= 




CU 
> 
CD 



> CO 

if a 

CO '<o 



65 



EP 1 715 686 A1 




f 

U 
00 

o 




CD 
Q. 



O 
CD 
X5 



-type 


type 


CD 
C3 

1 


I 


1 






c 

m wmmm 




O 










_str 


lica 


eco 


Clip. 


app 


i 

CO 



O 
Q 



TO 
o 

to 



CD 
o 



I 



CX3 
CD 




O 



CD 



"to 

LU 

\ 



O 

I 

CD 



CO 
CD 



CD 

I 

O 



I 

a. 

LLl 



o 

! 

CD 



O 
to 
a? 

CO 

^1 

TO 

to 

^1 

— 



TO 

2i 
-•— ' 

to 

1 

a> 

O 

o 

! 



CD 



^1 

TO 
CD 



in 



I 

CD 



CD 
Q_ 



£ 
TO 
CD 

^ 1 

CO 

I 

Q. 

yj 



i 

<D 



to 

43 

CD 



.5^ 

LU 



O 



CD 

O 



t 

CD 



CO 
CD 



C 

I 

O 



^1 

3 



I 

CD 



to 
to 

TO^ 

to 
Q 



TO 
CD 



to 

t 

CD 
O 



TO* 

_ i 







1 ■"! 






1 

CD 


o 
















51 






I 






E 


TO 


• • « 


CO 


CD 




CD 








to 




to 


1 




1 


CD 




CD 








O 




O 






1 

V— 


o 




O 

M— 


1 




1 






Cl. 


TO 




TO 








^1 




1 






LU 



00 
CO 

■ 

lZ 



U 

N 
N 
N 
N 
N 























c 






U 


E 




Cl. 


Sequen 


TO 


u 


u 


Prog 





66 



EP 1 715 686 A1 




67 



EP 1 715 686 A1 



o 



E 
P 



cu 



CD 



o 



TO < 



F < 



CD 



CO 
O 

o 

O 
>- 



Ct5 

X 



'CD 



m 











O 




c 








CO 




u 








CO 






+-» 




CO 


A A 



CD 

I 



£ 



A" 



cu 



I 

CI. 
3 



CD 

o 

CD 



CO 



X3 



JO 

I 

u 

CO 



CD 



CL 



CO 

I 

I » 

CO 

CO 

1 

u 

c; 

CO 



o 



c/) 

o 
qI 



CD 
Nj 

CO 



CD 










CL 
>x 








ST 






CD 




CD 


J 










03 


• • • • 


CO 


• • 


lay 










a. 


-Q 








-Q 


3 








3 


CO 




CO 




CO 



CD 



j5 



3 

CO 



00 
iS >- 



n 
"co 



* • 



CO 
CO 

Q. 



CO 
Q. 

CO 



• • • • 



CO 

a. 

3 
CO 



CO 
Q. 

3 

CO 

I 



68 



EP 1 715 686 A1 




CD 

II 



o 



W 

CD 

I 

a. 



I 



CO 



CD 

li 

CO 
4-' 

I 

t » I 

I 

(JO 



z> 



II 



I UJ 









» 


II 


II 






03 


cn 






CO 


CO 


1 


1 




LU^ 




CO 


a. 







! II 



i 
I 
1 
I 
I 



il 

CO 



CO 



85 



H 

CO 
CO 



CO 



il 



CO 


CO 




-1— • 


II 


II 








CO 


TO 


CO 




I 


co^ 










^1 


CO 




1— 


85 





ii 



II 


"IT 

V — 


CO 


CD 




CO 

1 


\ 

CL. 
LXJ 


2j 


1 

Ss 





CD o 
1 (X5 



I ^1^^ —3 



ql 



o 




II 



. o 

, Ou. 

; 2^ 













^< — 


il 


ii 




Izf 




CCS 




+— » 


CO 


00 


1 




Ill 

} 




00 




H- 







CD 
=«: 
i! 

C 

o 

^^^^ 



1 LU 





<^ 






1! 


II 


t: 




CO 




CO 

1 


CO 


C3u. 
1 i 1 


1 

CL. 

1 t i 


co' 


1 








&5 



I 
I 
I 
I 
i 
I 
I 
I 



II 



Cl, 

s 

4— ' 

LLI 







! 
! 






! 


"If 


II 


} 






} 


CO 


CO 


1 


00 

I 


CO 


I 




a-' 




UJ 


^, ,1 1 


» 
1 


1 

CO 


1 


( 






1 




00 


1 

1 



II 

4^ 





LO 






il 


il 






s 




CO 


00 

1 


CO 




1 

Cl- 


UJ 


UJ 


1 


I 


CO 






CL. 


CO 



CO 


00 


CO 


4t 
II 


jf 


il 


CO 






CO 




CO 


CO 




1 


r\ 




LU 


UJ 




1 




4-* 


CO 


^1 








UJ 




CO 



-9» 



=14: 
ii 



O 







4-» 






il 

4-» 








c5 


CO 






cry 

1 


00 




Cu 


UJ 

1 




CO 


1 


qI 







' "^ "^ 



to 



■CO 







L_ 


0) 0) 






>^ 




ex. 5~C 


ow 

L/ w ¥ 






t„l,,i y 


















0 
1 






r— - ^_ 



69 



EP 1 715 686 A1 




70 



EP 1 715 686 A1 



CO 



a. 

a 
>^ 

CO 

a 

Q 



U 

c 

CO 

a 
E 

Q 



CD 

c 

CO 

! 



o 



CO 

E 
o 

~\ 

Q- 
O 



A — A 




3 
_ 

> 

CU 

c 

^4""" iwKI"* ^^^^^ 

cu S ^ 

>*/ CO Hi 

^ C ^ CP 
C >,.^ CL 

3 Q E O 



M — 
O 

cn 
cu 



O 



as 

o 
o 

>- 
"a 

CD 
X 



■o 

CO 



_Time 


cu 




_Tim 


1 




tem 






1 


o 


em 




Playl 










i 






a 


o 


bPa 


Syn 


CO 











E 

cu 

j 1 

JO 



o 

I 

CO 
CL 

CO 
CO 



u 

CO 



o 



CO 

O 

{ 



CU 
N 

CO 

I 

a. 



CU 






CL 






>^ 






1 




J J 


ath_ 


• • • • 


ayll 












o 












CO 



E 



JO 

CO 

I 

c7) 



c/o 



CO :z 

JO >I 



CO 

c 

'co 



CO 



CO 



CO 



• « 



CO 

cx 

jQ 
15 
CO 



• • • 



CO 
Q. 



CO 

a. 

o 
CO 



71 



EP 1 715 686 A1 



O 

LO 



LO 



"O 



1 ID 
CO 



CD 
CO 



CCS 



~ o 

t o 

















ve 








o 















1 




i 

CD 




> 


CO 


Con 



00 



CX) 



O CCD C 




CO 



CO 

o 



o 



72 



EP 1 715 686 A1 



FIG. 45 



Jump Playback from Arbitrary 
Point on Playltem Timeline 





f ^ — ' S 1 


Calculate Offset a indicating Point on Playltem Timeline 










Convert Offset a to Coordinate on 
MainClip Timeline (ln_time + Offset a ) 








/ , — ' 


S3 


Convert Coordinate on MainClip Timeline (ln_time 
+ Offset ar ) to Address oc using Main Clip EP_map 







S31 



V 

Go to Step 
S4 in FIG- 30 



SubClip EP_map 
= 1 :Vicleo Type? 



YES 



S32 



Convert Offset a to Offset /? of Coordinate 

on SubPlaylTem Timeline 

(= Offset a - Sync__Start_PTS_of__Playltem) 



S3 3 



Convert Offset /3 to Coordinate on SubClip Timeline 
(= SubPlayltem_ln_Time + Offset (i ) 



S34 



Convert SubPlayltem„in_Time + Offset /? 
to Address /? of Point in SubClip 

using EP_map 



S3 5 



Simultaneously Read BD-ROM Starting from 
GOP of Primary Video at Address a and 
Local Storage Starting from GOP of 
Secondary Video at Address /? 



C END^ 



73 



EP 1 715 686 A1 



FIG. 46 

Closest Entry Point 



Upper 
1 St Level: 
TS Packets 



SPN=n1 SPN-n2 SPN=n3 SPN=n4 SP N=n5 ^PN^ nS 



Upper 
2nd Level: 
EP_map 









— - 






Entry Point=#1 


Entry Point=#3 


Entry Point=#5 


SPN_EP_Start=t1 


SPN_EP_Start=t3 


SPN_EP_Start=t5 


PTS EP Start=n1 


PTS_EP_Start=n3 


PTS_EP_Start=n5 



I 



Upper 3rd Level: 
SubClip Timeline 



V 



Entry Point=#2 



SPN EP Start=t2 



PTS_EP_Start=n2 



Entry Point=#4 



SPN EP Start=t4 



PTS EP Start=n4 



V 



Entry Point=#6 



SPN EP Start=t6 



SubPlayltem_ln_time 



Sync_Start_Pts_of_Play Item 
Upper 4th Level: SubPlayltem 



Offset/? 



PTS_EP_Start=n6 

t 



TM 



Timeline ; 



Offset/? 



Middle Level: Playltem Timeline 



Offset 



Lower 3rd Level: 



In time 





Entry Point=#1 



Lower 
2nd Level: 

EP„map 



PTS. 


_EP. 


_Start= 


=t1 


SPN. 


.EP. 


.Start= 


=n1 



V 




Offset a 



Entry Point=#3 



- 

Entry Point=#5 



PTS EP Start=t3 



SPN EP Start=n3 



Entry Point=#2 



PTS. 


EP 


.Start=t2 


SPN. 


.EP. 


.Start=n2 



V 



PTS EP Start=t5 



SPN EP Start=n5 



Entry Point=#4 



PTS EP Start=t4 



SPN EP Start=n4 



Entry Point=#6 



PTS EP Start=t6 



SPN_EP_Start=n6 

t 



SPN=n1 SPN=n2 SPN=n3 SPN=n4 SPN=n5 SPN=n6 



Lower 
1 St Level: 
TS Packets 



m 



'A 



/j 



Closest Entry Point 




74 



EP 1 715 686 A1 



FIG. 47A 



Primary 

Video Playback 

Secondary 
Video Playback 



Lock Operation 



FIG. 47B 



Primary 

Video Playback 

Secondary 
Video Playback 



Unlock 
— ?— 



:3 



Unlock 



No Lock 



FIG. 47C Lock Operation 

Primary 



Video Playback 

Secondary 
Video Playback 



75 



EP 1 715 686 A1 



INTERNATIONAL SEARCH REPORT 



Inlenuitiontil applicalion No. 

PCT/JP2 00 5/016640 



A. CT.ASSTFTCATTON OF SUBJECT MATTER 

H04N5/93 {2006 ,01) , G11B20/10 {2006 . 01) , G11B20/12 {20 06 , 01) , G11M27/00 
(2006 .01) , H04N5/85{2QQ6 , 01) , H04N5/92 {2006 .01) 



Accotrli tie. to Tiiterfintiotinl PntetiT < 'Inssificntinn f TPO or to botli untionnl dnssificntion nnd TPC 



B. FIELDS SE^^CIIED 



Minimum documentation searched (classification system followed by classification sjanbols) 

H04N5/93 {2006 .01) , GllB20/l6 {2006 . 01) ] G11B20/12 {20 06 , 01) , G11B27/00 
(2006 . 01) , H04N5/85{2006 . 01) , H04N5/92 {2006 . 01) 



Documentation searched other than minimum documentation to the extent that such documents are included id the fields searched 
Jitsuyo Shiran Koho 1922-1996 Jitsuyo Shinan Toroku Koho 1996-2005 

Kokai Jitsuyo Shinan Koho 1971-2005 "Itoroku Jitsuyo Shinan Koho 1994-2005 

Electronic data base consulted during the iatemational search (name of data base and, where practicable, search terms used) 



C. DOCUMENTS CONSIDERED TO BE RETEVANT 



Category^ 



Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



X 
Y 



JP 2003-68057 A (LG Electronics Inc.), 
07 March, 2003 (07.03.03), 
Par. Nos. [0011] to [0020] 

JP 8-191423 A (Mitsubishi Electric Corp.), 
23 July, 1996 (23.07.96), 
Par. Nos. [0054] to [0061] 

JP 11-75159 A (Toshiba Corp.), 
16 March, 1999 (16.03.99), 
Par. Nos. [0078] to [0085] 



1-4,8-10, 

16-17 
5-7 , 11-15 

5-7, 11-15 



1-17 



I I Furtlier documents are listed in the continuation of Box C. 



I ^ I See patent family annex. 



* Special categories of cited documeiits: "X" 

'A" document defining the general state of the art which is not considered 
to be of particular relevance 

*E" earlier JipplicMJuin tirpaJenl. but published t)n t>r a.lll.er l.he inlemfilionjil "X" 
filing date 

"L" document whicli may throw doubts on priority claim(s) or wliich is 

cited to establish the publication date of another citation or other «y" 
special reason (as specified) 

'O' document referring to an oral disclosure, use, exhibition or other means 

"P" document published prior to the international filing date but later than 

the priority date claimed "& " 



later document published after the international filing date or priority 
date and not in conflict with the application but cited to understand 
the principle or theory underlying the invention 

document of particular relevance; the claimed invention cannot be 
considered novel or cannot be considered to involve an inventive 
step when the document is taken alone 

document of particular relevance; the cJaimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 



Date of the actual completion of the international search 

06 December, 2005 (06.12.05) 


Date of mailing of the international search report 

20 December, 2005 (20.12.05) 


Name and mailiiig address of the ISA*^ 

Japanese Patent Office 

Facsimile No. 


Authorized officer 
Telephone No. 



Form PCT/ISA/'210 (second sheet) (April 2005) 



76 



EP 1 715 686 A1 



UNTERNATIONAI. SEARCH REPORT 

Information on patent family members 



International application No. 

PCT/iJP2005/016640 



JP 2003-68057 A 



JP 8-191423 A 



JP 11-75159 A 



2003 , 03 . 07 



1996,07,23 



1999.03.16 



(Family : 


none) 




(Family: 


none) 




AT 


00018 


2026 


T 




CA 


00217 


3812 


A 




DE 


06960 


3118 


C 




EP 


00073 


7975 


A2 




KR 


00020 


9847 


B 




KR 


00025 


3614 


B 




KR 


00025 


6517 


B 




KR 


00025 


66 63 


B 




US 


00611 


2009 


Al 




US 


00613 


4383 


Al 




US 


2002/ 


0090206 


Al 


US 


2002/ 


0191963 


Al 


US 


2003/ 


0002858 


Al 



Form PCT/IS^\/210 (pateat family aimex) (April 2005) 



77 



EP 1 715 686 A1 

REFERENCES CITED IN THE DESCRIPTION 

This list of references cited by the applicant is for the reader's convenience only. It does not form part of the European 
patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be 
excluded and the EPO disclaims all liability in this regard. 

Patent documents cited In the description 

• JP 2002247526 A [0003] 



78 



